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1.0.  INTRODUCTION 


As  a  major  producer  of  army  materiel,  the  U.S.  Army  Tank-Automotive 
Command  (TACOM)  must  continually  evaluate  its  production  processes, 
techniques,  and  equipment  to  insure  that  high-quality  TACOM 
commodities  are  produced  in  a  timely  and  economical  manner  in  order  to 
best  support  the  Army  Industrial  Preparedness  Program.  This 
evaluation  provides  TACOM  numerous  opportunities  to  participate  in  the 
Army'  Manufacturing  Technology  (MANTECH)  Program,  which  is  designed  to 
advance  the  manufacturing  state  of  the  art,  modernize  current 
operations,  and  comply  with  regulatory  requirements.  However,  because 
these  opportunities  normally  exceed  the  financial  resources  available 
to  fund  such  projects,  TACOM  must  exercise  sound  judgment  in  the 
determination  of  which  projects  to  sponsor. 

The  purpose  of  this  report  is  to  develop  a  methodology  for  making  this 
selection  within  TACOM.  The  nature  of  the  TACOM  Manufacturing 
Technology  Program,  the  typical  flow  that  results  in  project 
recommendation,  and  the  process  of  project  evaluation  are  addressed; 
recommendations  designed  to  improve  the  process  within  TACOM  are 
presented. 

A  method  for  determining  the  economic  rank  of  MANTECH  projects  is 
provided  as  Appendix  A.  The  evaluation  parameters  that  are  used  to 
determine  the  value  of  a  proposed  MANTECH  project  are  defined  and 
discussed  in  Appendix  B.  The  Interim  Progress  Report  for  the  MANTECH 
Prioritization  Methodology  Program  is  provided  as  Appendix  C  for 
background  information. 

2.0.  BACKGROUND 

The  Manufacturing  Technology  Program  is  a  joint  service  program.  In 
accordance  with  Department  of  Defense  (DOD)  Instruction  4200.15 
(Ref.  1),  manufacturing  technology  refers  to: 

...any  action  undertaken  which  has  as  its  objective 
(1)  the  timely  establishment  or  improvement  of  the 
manufacturing  processes,  techniques,  or  equipment 
required  to  support  current  and  projected  programs, 
and  (2)  the  assurance  of  the  ability  to  produce, 
reduce  lead  time,  insure  economic  availability  of 
end  items,  reduce  costs,  increase  efficiency, 
improve  reliability,  or  to  enhance  safety  and  anti¬ 
pollution  measures  as  covered  in  DOD  Directive 
6050.1. 

Also  in  accordance  with  this  instruction,  a  particular  Manufacturing 
Technology  Project  refers  to: 

...the  development  or  improvement  of  manufacturing 
processes,  techniques,  and  equipment  by  the 
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Government  or  private  industry  to  provide  for 
timely,  reliable,  economical  manufacture  of 
Department  of  Defense  materiel.  The  objective  is 
to  bridge  the  gap  between  feasibility  and  full- 
scale  production  and  to  achieve  parity  between 
manufacturing  technology  and  research  and  develop¬ 
ment  advances  which  will  smooth  the  translation  of 
systems  design  criteria  into  reliable  production 
hardware.  Manufacturing  technology  projects  may 
also  provide  engineering  support  to  the  modern¬ 
ization  of  the  industrial  production  base  to 
provide  for  improved  capability  to  meet  a  military 
contingency.  They  are  normally  broad-based  in 
application,  are  production  oriented  even  when  they 
are  performed  in  a  prototype  environment,  and  are  , 
expected  to  result  in  a  practical  process  for 
production.  They  may  include  the  application  of 
new  or  improved  techniques  or  equipment  to 
manufacture  specific  weapon  systems,  components, 
end  items,  and  prototypes,... 

As  stated  in  a  joint  “Statement  of  Principles  for  Department  of 
Defense  Manufacturing  Technology  Program,"  the  objectives  of  this 
program  are  to: 

§  Aid  in  insuring  the  economical  production  of  qualitatively 
superior  weapon  systems  on  a  timely  basis; 

•  Insure  that  advanced  manufacturing  processes,  techniques,  and 
equipment  are  used  to  reduce  DOD  material  acquisition  costs; 

•  Continuously  advance  manufacturing  technology  to  bridge  the  gap 
from  research  and  development  (R&D)  advances  to  full-scale 
production; 

•  Foster  greater  use  of  computer  technology  in  all  elements  of 
manufacturing; 

•  Insure  that  more  effective  industrial  innovation  is  stimulated 
by  reducing  the  cost  and  risk  of  advancing  and  applying  new  and 
improved  manufacturing  technology;  and 

•  Insure  that  manufacturing  processes  are  consistent  with  safety 

and  environment  considerations  and  energy  conservation 

objectives. 

3.0.  MANUFACTURING  TECHNOLOGY  PROGRAM  ORGANIZATION 

The  U.S.  Department  of  the  Army  (DA)  has  an  established  Manufacturing 
Technology  Program.  This  program  is  managed  and  controlled  by  the 
Directorate  of  Manufacturing  Technology  (DMT)  within  Headquarters, 
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U.S.  Army  Material  Development  and  Readiness  Command  (HQ-DARCOM). 
Technical  support  is  provided  to  DARCOM  by  the  Industrial  Base 
Engineering  Activity  (IBEA)  and  by  the  Army  Materials  and  Mechanics 
Research  Center  (AMMRC).  In  addition,  the  Army  participates  in  a 
tri service  Manufacturing  Technology  Advisory  Group  (MTAG)  to 
facilitate  coordination  of  the  diverse  activities  of  the  three 
military  services  (Army,  Navy,  and  Air  Force).  Figure  3-1  illustrates 
the  management  organization  for  the  Manufacturing  Technology  Program. 

As  a  Major  Subordinate  Command  (SUBMACOM),  TACOM  is  responsible  for 
managing  its  own  Manufacturing  Technology  Program.  Within  TACOM,  this 
program  is  part  of  the  DA  Production  Base  Support  Program  (PBSP),  and 
the  responsibility  for  it  resides  with  the  Tank-Automotive  Systems 
Laboratory,  Metals/Welding  (DRSTA  RCKM).  This  program  applies  to  both 
combat  and  tactical  vehicles.  Within  the  Systems  Laboratory,  the 
program  has  been  assigned  the  broad  title  of  Manufacturing  Technology 
Program,  and  it  is  conducted  under  the  Production  Engineering  Measures 
(PEM)  category  of  Army  projects,  which  includes  both  Military 
Adaptation  of  Commercial  Items  (MACI)  and  Manufacturing  Methods  and 
Technology  (MMT)  projects. 

4.0.  MANTECH  PROGRAM  CYCLE 

The  process  for  planning  MANTECH  projects  from  concept  through 
execution  is  highly  structured  and  involves  a  number  of  review  and 
evaluation  steps  prior  to  approval.  In  addition  to  representatives  of 
DA  who  give  final  approval,  IBEA,  AMMRC,  DMT,  DARCOM,  and  each 
SUBMACOM  are  all  active  participants  in  the  planning  process.  The 
three  major  activities  of  the  planning  process  are  the  5-year  program 
plan,  the  budget  review,  and  the  apportionment  review.  Milestones  for 
each  activity  are  illustrated  in  Figure  4-1;  the  functional  flow  shown 
(for  a  single  project)  is  that  adopted  by  DARCOM  for  FY  1984.  Details 
of  this  functional  flow  can  be  expected  to  be  changed  from  time  to 
time.  However,  the  principal  steps  and  responsibilities  are  generally 
retained. 

4.1.  Five-Year  Program  Plan. 

Each  SUBMACOM  maintains  a  5-year  program  plan  that  is  updated 
annually.  The  plan  contains  a  summary  of  all  projects  planned  to  be 
executed  within  the  next  5  years.  During  May,  DARCOM  (IBEA)  issues  a 
call  letter  to  each  SUBMACOM  requesting  that  an  update  of  the  program 
plan  be  submitted  by  June.  The  program  plan  contains  cost  estimates, 
a  description  of  the  problem,  and  the  proposed  solution  for  each 
project.  IBEA  collates  these  plans  by  mission  and,  if  necessary,  by 
manufacturing  specialties.  Each  SUBMACOM  has,  presumably,  an 
established  set  of  unique  missions.  TACOM,  missions  are: 

•  Armor, 

•  Drive  Train, 

•  Body/Frame, 
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Figure  3-1.  Manufacturing  Technology  Program  Organization 
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•  Track, 

t  Suspension,  and 

•  General. 


The  call  letter  also  includes  funding  guidelines  for  each  year  of  the 
5-year  period. 

After  receiving  and  reviewing  these  plans,  IBEA  consolidates  them  into 
one  report  representing  the  MANTECH  effort  for  all  of  DARCOM.  In 
October,  this  report  is  distributed  to  each  SUBMACOM  and  to  MTAG, 
among  others. 

4.2.  Budget  Review. 

The  conduct  of  reviews  and  evaluations  of  new  and  ongoing  projects  and 
the  preparation  of  documents  to  support  the  budget  review  is  the  next 
step  in  the  planning  process.  The  sources  of  new  candidate  projects 
include  both  government  and  industry,  but  usually  arise  from  the 

manufacturing  community.  Obviously,  those  projects  contained  in  the 
5-year  program  plan  are  an  initial  input  to  the  process.  However, 
many  projects  planned  for  the  outyears  fall  by  the  wayside,  and  new 

projects  are  submitted  in  their  place.  Ongoing  projects  of  previous 

years  are  reviewed  each  year  in  the  same  manner  as  new  projects. 

The  budget  review  process  also  begins  with  a  call  letter  issued  by 
DARCOM  (IBEA)  to  each  SUBMACOM  early  in  the  calendar  year.  This 
letter  requests  the  preparation  of  Part  I  of  the  P-16  form 

(Appendix  C)  for  each  proposed  project.  This  form  is  to  be  submitted 
no  later  than  the  first  week  of  May.  These  proposals  are  for  projects 
to  be  initiated  2  (fiscal)  years  later.  Usually,  these  requests  are 
relayed  by  the  MANTECH  office  of  the  SUBMACOM  to  other  organizations 
within  their  purview  for  action.  The  MANTECH  office  responsible  for 
fulfilling  the  IBEA  request  sets  its  own  schedule  for  the  review, 
evaluation,  and  selection  of  projects  to  be  submitted  before  the  May 
deadline.  In  some  cases,  the  identification  and  selection  of 
candidate  projects  is  a  year-long  activity. 

Following  submittal  of  the  Part  I  P-16  form,  IBEA  (and  AMMRC)  conduct 
what  is  called  a  working  review;  usually  beginning  in  March  and 
extending  through  May,  dependent  on  SUBMACOM  response  time.  During 
this  review,  representatives  of  IBEA  meet  with  those  of  SUBMACOM  to 
discuss  issues  and  to  resolve  specific  questions  regarding  the 
information  contained  in  the  P-16  form. 

As  a  result  of  the  working  review,  the  SUBMACOM  revises  individual 
project  P-16  forms  as  necessary  and  submits  a  second  data  package  to 
IBEA/ AMMRC  no  later  than  June  in  support  of  the  "official  review." 
For  this  review,  the  total  cost  of  all  projects  submitted  should  not 
be  greater  than  125  percent  of  the  budget  guidelines  given  in  the  call 
letter.  During  the  official  review,  both  IBEA  and  AMMRC  prepare  an 
independent  evaluation  document  that  is  forwarded  to  both  DMT  and 
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SUBMACOM  in  support  of  the  final  budget  review.  Each  document 
contains  technical  comments,  scores  each  proposal,  and  recommends  the 
disposition  of  each  project.  (IBEA  and  AMMRC  use  different  methods  of 
scoring,  see  Section  2. 1.4. 2.  of  Appendix  C.)  In  addition  to  approval 
or  disapproval,  three  other  kinds  of  disposition  can  be  recommended: 
(1)  approve  pending  rewrite  (A/R),  which  means  the  proposal  is 
technically  worthwhile  but  minor  changes  should  be  made  to  the 
documentation  for  better  clarity  or  format;  (2)  defer  for  rewrite 
(D/R),  which  means  the  proposal  has  promise  but  the  documentation  is 
inadequate  and  requires  major  rewrite;  and  (3)  defer  for  technical 
reevaluation  (D/TR),  which  means  the  concept  is  good  but  there  are 
technical  problems  to  be  resolved  and  a  rejustification  is  required. 

Following  the  official  review  by  IBEA  and  AMMRC,  the  budget  review  is 
conducted  in  early  July.  The  review  is  chaired  by  DMT  with 
representatives  of  DA,  IBEA,  AMMRC,  and  SUBMACOM  in  attendance.  At 
this  meeting,  final  decisions  are  reached  regarding  MANTECH  projects 
proposed  by  SUBMACOM.  If  a  project  is  approved,  SUBMACOM  then 
schedules  preparation  of  Part  II  of  the  P-16  form  in  support  of  the 
apportionment  review,  normally  held  the  following  April.  If  the 
project  receives  a  contingent  recommendation  (i.e.,  A/R,  D/R,  or 
D/TR),  the  submittal  to  DMT  of  a  revised  Part  I  P-16  form  may  be 
required  within  8  to  10  days,  or  consideration  of  the  proposal  may  be 
deferred  until  the  following  year. 

4.3.  Apportionment  Review. 

The  apportionment  review  is  the  final  step  of  the  planning  process. 
It  begins  officially  with  a  call  letter  issued  by  DARCOM  (IBEA)  to 
each  SUBMACOM,  normally  in  January,  requesting  preparation  of  Part  II 
of  the  P-16  form  (Appendix  C)  for  each  project  that  survived  the 
budget  review  and  is  scheduled  for  procurement  the  following  fiscal 
year.  Although  a  parallel  process  in  support  of  the  budget  review  for 
new  projects  is  occurring  during  the  same  period,  the  apportionment 
review  is  held  earlier  than  the  budget  review,  allowing  procurement  to 
begin  early  in  the  following  fiscal  year. 

The  apportionment  review  is  the  same  as  the  budget  review,  except  that 
no  working  review  is  held.  The  same  organizations  are  present  at  this 
review  as  were  present  at  the  budget  review,  and  final  decisions  are 
reached  as  to  which  projects  are  approved  for  funding. 

5.0.  TECHNICAL  DISCUSSION 

The  following  discussion  presents  the  results  of  the  study  to  develop 
a  methodology  consistent  with  the  planning  and  background  information. 

5.1.  Study  Approach. 

5.1.1.  Review  of  Regulations.  Applicable  DOD  and  DA  regulations  and 
other  related  documents  were  analyzed  to  determine  the  goals  and 
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objectives  of  manufacturing  technology  programs  as  they  might  relate 
to  the  TACOM  MANTECH  Program.  These  documents  also  were  used  to 
formulate  the  major  evaluation  categories  and  subordinate  evaluation 
parameters  defined  in  this  report.  Although  a  composite  listing  of 
all  criteria  contained  in  these  documents  is  quite  exhaustive,  it  was 
found  that  there  is  a  great  degree  of  commonality  among  them.  Based 
on  this  analysis,  it  was  determined  that  the  TACOM  MANTECH 
prioritization  methodology  can  be  constructed  from  10  evaluation 
parameters  that  can  be  grouped  conveniently  according  to  three 
categories;  need,  risk,  and  economic  benefit.  Further  discussion  of 
this  approach  is  contained  in  Section  5.3.  More  detailed  information 
on  parameters  contained  in  the  applicable  documents  is  given  in  the 
Interim  Report  (Appendix  C). 

5.1.2.  Survey  of  Other  Commands.  As  discussed  in  the  Interim  Report, 
Appendix  C,  visits  were  made  to  eight  military  organizations,  other 
than  TACOM,  to  assess  the  processes  they  use  to  select  and  prioritize 
MANTECH  projects.  A  common  element  found  was  that  all  organizations 
generally  rely  on  a  "bottom-up"  approach  to  originating  projects, 
which  is  not  unexpected  because  the  government  working-level  personnel 
are  most  familiar  with  production  problems  and  are  closer  to  their 
industry  counterpart  where  many  of  the  improvements  are  generated. 
However,  approximately  50  percent  of  these  organizations  attempt  to 
define  major  thrust  areas,  convene  guidance  conferences,  or  use  cost 
driver  analyses  to  focus  on  important  areas.  All  organizations  do 
consider  economic  benefit,  although  it  is  not  an  overriding 
consideration  in  the  prioritization  of  projects. 

None  of  these  organizations  use  a  formal  method  of  numerically 
weighting  or  ranking  parameters  to  define  priorities.  Although 
several  have  attempted  such  an  approach,  no  successful  system  has  been 
found  because  there  is  a  wide  variation  in  project  types.  As  a 
substitute,  all  organizations  rely  heavily  on  a  peer  group  review 
process  using  the  technical  expertise  within  their  MANTECH  group, 
assisted  in  a  matrix  fashion  by  other  specialists  within  other  parts 
of  the  organization.  TACOM  does  not  currently  do  this.  Many  of  these 
organizations  also  have  developed  a  standard  operating  procedure  or 
internal  regulation  document  to  guide  their  process.  TACOM  does  not 
have  such  a  document. 

5.1.3.  Analysis  of  P-16  Exhibits.  TACOM  P-16  forms  were  reviewed  for 
MACI  and  MMT  projects  pertaining  to  weapons  and  tracked  combat 
vehicles  and  tactical  vehicles  for  FY  1983,  1984,  and  1985.  The 
purpose  of  this  review  was  to  understand  the  type  and  depth  of 
information  available  to  the  project  evaluator(s)  and  to  assist  in 
deriving  a  suitable  prioritization  methodology  for  MANTECH  projects. 
The  primary  results  of  this  analysis  are  discussed  below. 

Projects  vary  widely  in  technical  scope.  Some  projects,  such  as 
Industrial  Productivity  Improvements,  are  quite  broad  and  may,  for 
example,  apply  to  all  the  operations  performed  at  a  given  depot. 
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others,  while  perhaps  written  for  a  complete  commodity  end  item  (e.g., 
Abrams  (Ml)  combat  vehicle),  may  contain  as  many  as  11  specific 
tasks.  The  project  evaluation  methodology  must,  therefore,  be 
flexible.  This  flexibility  was  taken  into  consideration  in  developing 
the  categories,  parameters,  and  prioritization  algorithm  discussed  in 
this  report. 

The  majority  of  the  projects  are  multiyear.  More  than  50  percent  of 
FY  1984  projects  were  carryovers  from  FY  1983.  A  similar  comparison 
can  be  made  for  FY  1985  and  FY  1984.  These  carryovers  may  be  due  to 
funding  limitations  that  restricted  new  starts.  However,  because  this 
practice  exists,  the  project  prioritization  algorithm  developed  in 
this  report  was  structured  to  take  it  into  account. 

MACI  projects  are  not  in  the  same  class  as  MMT  projects.  However, 
they  represent  a  rather  small  percentage,  i.e.,  approximately  10 
percent  of  the  projects,  and  it  is  believed  that  the  methodology 
developed  in  this  report  can  accommodate  them. 

"Productivity/Cost  Savings"  is  the  predominant  benefit  associated  with 
projects.  However,  there  is  a  large  range  of  additional  benefits  that 
are  often  cited.  For  this  reason,  the  methodology  constructed  in  this 
report  takes  into  consideration  both  primary  and  secondary  benefits. 

The  information  provided  is  generally  adequate  for  a  TACOM  evaluator 
to  understand  the  nature  of  the  project  and  what  the  project  is  to 
accomplish.  However,  because  TACOM  has  not  developed  a  prioritized 
listing  of  needs,  it  is  impossible  to  assess  the  importance  of  a 
particular  project  in  relationship  to  others  being  considered. 
Although  this  information  may  not  need  to  appear  on  the  P-16  exhibit, 
it  is  considered  essential  to  the  evaluation.  This  feature  is 
developed  in  some  detail  in  this  report. 

In  many  cases,  the  depth  of  the  economic  analysis  also  varies  as  a 
function  of  the  technical  definition  of  the  project.  Some  projects 
may  be  divided  into  specific  subtasks  where  detailed  cost  information 
is  available  based  on  actual  man-hours  and  types  of  operations.  Other 
projects  may  be  so  broad  in  nature  that  only  a  percentage  estimate  of 
savings  can  be  made  based  on  the  total  cost  of  the  operation  as  it  is 
now  performed.  Return  on  investment  (ROI)  data  may  vary  from 
just-profitable  to  more  than  50  times.  Due  to  this  variance,  and  in 
consideration  of  the  subjectivity  that  goes  into  arriving  at  the 
numbers,  the  prioritization  algorithm  described  in  this  report  was 
constructed  to  place  emphasis  on  benefits  and  costs,  while  not 
allowing  wide  swings  in  ROI  to  potentially  bias  the  results. 

In  addition,  it  is  not  clear  from  either  the  P-16s  reviewed  or  the 
outline  for  Part  II  of  the  P-16s  that  the  proper  cost  and  savings 
values  are  included.  The  recurring  savings  column  should  be  a  net 
savings  (benefits  minus  cost)  including  any  operation  and  maintenance 
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costs  incurred  during  the  project  life.  Also,  implementation  costs 
should  include  not  only  implementing  the  MANTECH  project,  but 
implementing  the  MANTECH  proven  concept  into  mass  production.  This 
may  require  additional  nonrecurring  capital  improvement  costs 
associated  with  facility  modifications,  interruptions  in  the  current 
manufacturing  process,  publication  of  new  procedures  or 
specifications,  etc.  Clarification  of  these  points  will  insure  that 
all  projects  are  analyzed  in  a  consistent  manner. 

5.2.  Typical  Functional  Flow. 

To  have  an  effective  prioritization  methodology,  it  is  considered 
necessary  for  the  entire  MANTECH  Program  to  be  reviewed  as  an  ongoing 
process,  from  initial  planning  through  project  implementation  and 
follow-up.  The  evaluation  of  proposed  projects,  and  their  subsequent 
ranking,  then  becomes  an  integral  part  of  this  process,  as  opposed  to 
some  distinct  entity.  Because  TACOM  has  no  formally  established 
functional  flow  for  its  MANTECH  Program,  a  typical,  closed-loop, 
iterative  process  was  developed  as  an  aid  for  developing  an  overall 
prioritization  methodology.  This  process,  which  can  be  applied  to 
TACOM,  is  shown  in  Figure  5-1  and  discussed  below. 

5.2.1.  Monitor  Commodity  Manufacture.  Monitoring  of  commodity 
manufacturing  is  essential  in  initiating  the  process  of  identifying 
and,  ultimately,  prioritizing  candidate  MANTECH  projects.  Cost  and 
schedule  data,  safety,  and  other  manufacturing  problems  in  general 
must  be  monitored.  This  activity  occurs  early  in  the  planning  process 
and  involves  a  rather  broad  participation.  DMT  (HQ-DARCOM)  can 
conduct  technology  assessments  and  is  familiar  with  future 
requirements  and  state-of-the-art  manufacturing  technologies.  This 
facilitates  DMT  preparation  of  guidance  packages,  call  letters,  and 
ultimately,  the  formulation  of  the  5-Year  Program  Plan.  Likewise, 
TACOM  MANTECH  representatives  and  Project  Managers  (PMs)  are  familiar 
with  current  and  future  potential  problems  that  can  be  solved  through 
the  MANTECH  Program.  Installation  personnel  (plants,  depots, 
laboratories)  are  closest  to  the  manufacturing  currently  in  process 
and  the  most  likely  to  identify  specific  projects  (box  (5)  of 
Figure  5-1)  and,  therefore,  can  provide  feedback  to  the  monitoring  of 
commodity  manufacture.  Each  organization  exercises  its  monitoring 
role  with  an  understanding  of  the  objectives  of  the  overall 
Manufacturing  Technology  Program. 

5.2.2.  Determine  Improvement  Areas.  The  results  of  monitoring 
conmodity  manufacturing  provide  the  ability  to  determine  broad 
technology  areas  where  the  application  of  MANTECH  Program  funds  might 
provide  the  greatest  benefit.  These  technology  areas  include  the 
classic  types,  e.g.,  metals,  nonmetals,  automation,  and  electronics. 
A  prioritization  of  these  areas  is  appropriate  at  this  point 

5.2.3.  Develop  Forcing  Functions.  The  development  of  forcing 
functions  enables  one  to  get  closer  to  actual  projects.  For  the 
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Figure  5-1.  Typical  MANTECH  Functional  Flow 


15 


commodity  being  considered,  the  technology  areas  identified  are 
examined  in  light  of  forcing  functions,  such  as  complexity,  high  cost, 
and  poor  quality.  This  identification  allows  one  to  start  framing  a 
more  specific  objective,  e.g.,  decrease  the  complexity  (forcing 
function)  in  welding  (technology  improvement)  tank  bodies 

(commodity).  To  insure  a  greater  probability  of  receiving  appropriate 
candidate  projects,  this  function  is  accomplished  with  a  full 

understanding  of  the  criteria  for  the  selection  of  MACI  and  MMT 

projects. 

5.2.4.  Communicate  Needs.  The  TACOM  MANTECH  office  should 

communicate  the  program  needs,  or  requirements,  to  the  installations 
in  order  to  provide  them  with  an  insight  to  the  types  of  projects  that 
are  considered  to  be  of  high  priority.  This  priority  designation  can 
be  accomplished  with  the  call  letter  for  P-16  exhibits  or  can  be 
coiranunicated  at  joint  seminars  and  conferences. 

5.2.5.  Identify  Candidate  Projects.  The  identification  of  candidate 

projects  initiates  the  P-16  flow,  which  is  normally  performed  by  the 
installation  that  originated  the  proposed  project.  As  discussed  in 
Section  5.2.1.,  installations  can,  however,  have  a  role  earlier  in  the 
process  through  the  development  of  technology  areas  and  forcing 
functions.  Again,  this  function  is  performed  in  recognition  of  the 

criteria  for  such  projects. 

5.2.6.  Screen  Projects.  Early  screening  of  candidate  projects  at  the 

installation  level  is  important  in  helping  to  eliminate  projects 
submitted  to  the  TACOM  MANTECH  office;  additional  screening  also  will 
be  performed  at  the  MANTECH  office  level.  This  function  is 

essentially  a  go/no-go  screen  against  the  established  criteria  for 
these  projects.  Those  criteria  include  (Ref.  2,  DARCOM  Handbook, 
defines  the  first  five  criteria): 

•  Military  Department  Requirement:  Every  project  must  satisfy  a 

current  or  anticipated  Military  Department  requirement  for 

which  manufacturing  techniques  are  needed.  Future  requirements 
normally  will  be  qualitative  in  nature  and  will  be  directed 
toward  the  adoption  of  manufacturing  technology  that  will 

increase  general  productivity. 

t  No  Duplication  of  Effort:  The  problem  solution  must  not  be 
otherwise  available  on  a  timely  basis  from  other  known  programs 
undertaken  with  government  support  or  private  capital. 

•  Manufacturing  Technology  Problem:  The  problem  solution  must 

require  the  establishment  of  new,  improved,  or  more  economical 
manufacturing  processes,  techniques,  or  equipment  rather  than 
basic  R&D-oriented  efforts  on  the  mere  application  of  existing 
processes,  techniques,  equipment,  or  facilities  for  the 

manufacture  of  specific  parts. 
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•  Adequate  State  of  the  Art:  Qualitative*  laboratory  type 
feasibility  of  the  effort  under  consideration  must  have  been 
demonstrated  sufficiently  by  experiment  or  extrapolation  of 
analytical  data. 

•  Worthwhile  Benefits:  Determination  must  be  made  that  a  project 
normally  will  result  in  one  or  more  of  the  following: 

--  Improved  responsiveness  to  current  and  projected 
requirements, 

—  More  effective  industrial  preparedness  base  that  will 
reflect  modern  manufacturing  techniques  and  result  in 
improved  defense  production  posture,  and 

--  Timely  manufacturing  technology  development  to  provide 
information  to  be  used  in  preparation  of  material 
specifications  or  contract  definitions  that  reflect  the 
most  advanced  manufacturing  state  of  the  art. 

•  High  likelihood  of  implementation  in  both  government  or 
industrial  facilities. 

•  Satisfy  a  stated  requirement  through  the  adaptation  of  a 
commercially  available  (domestic  or  foreign)  item,  generic 
component,  or  system. 

5.2.7.  Evaluate  and  Prioritize  Projects.  Evaluating  and  prioritizing 
candidate  projects  is  a  most  important  function.  This  function  is 
performed  by  applying  those  parameters  discussed  in  Section  5.3., 
using  the  methods  given  in  Sections  5.4.  and  5.5.  It  may  be  performed 
as  part  of  a  joint  evaluation  team  or  can  be  performed  exclusively 
within  the  higher  levels  of  TACOM.  At  this  point,  IBEA  can  be  brought 
into  the  process  so  the  evaluating  personnel  benefit  from  its  broader 
perspective  of  projects  being  pursued  within  other  SUBMACOMs. 

5.2.8.  Submission  for  Approval.  Candidate  project  suggestions  must  be 
submitted  to  DARCOM  for  final  approval.  This  submittal  occurs  in  two 
steps:  initial  submittal  in  July  as  part  of  the  budget  and  planning 
cycle  and  again  in  April  of  the  following  year  for  the  apportionment 
cycle  where  actual  funding  is  approved  for  specific  projects. 

5.2.9.  Conduct  Demonstration.  Following  project  approval  and  program 
and  fund  release,  TACOM  assumes  overall  responsibility  for  project 
execution.  Until  the  project  is  complete,  TACOM  will  provide  a 
MANTECH  Program  Project  Status  Report  (RCS-DRCMT-301 )  to  IBEA  and 
DMT.  These  reports  are  submitted  by  January  15  and  July  15  of  each 
year.  IBEA  reviews  the  project  status  and  makes  a  consolidated  report 
with  recommendations.  The  consolidated  report  is  forwarded  through 
HQ-DARCOM  to  Headquarters,  Department  of  the  Army  (HQDA). 
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In  addition  to  publishing  status  reports,  the  action  agency  is 
responsible  for  closely  monitoring  the  progress  of  the  project.  This 
monitoring  allows  for  a  reassessment  of  the  value  of  the  project  and 
the  opportunity  for  feedback  to  the  development  of  forcing  functions 
for  new  projects.  During  a  subsequent  year  evaluation  process,  an 
ongoing  project  may  be  determined  to  have  a  lower  evaluation  than 
newer  projects.  If  this  occurs,  and  funding  limitations  will  not 
allow  pursuing  both,  a  decision  may  have  to  be  made  as  to  which  one  to 
pursue.  This  situation  will  have  to  be  evaluated  on  a  case-by-case 
basis  taking  into  account  the  following  factors: 

•  Percent  of  total  planned  investment  expended; 

•  Success  to  date  in  terms  of  performance,  cost,  and  schedule; 

t  Cost  of  termination; 

•  Loss  of  benefits  if  project  is  not  completed;  and 

•  Impact  of  delaying  new  project. 

5.2.10.  Publish  Final  Results.  Upon  receiving  the  final  MANTECH 
Project  Status  Report,  IBEA  completes  a  Summary  Report  {RCS-DRCMT-302) 
highlighting  necessary  implementation  actions  and  possible  uses  by 
other  coninands  and  services.  This  report  is  disseminated  to  all 
activities  having  technical  interest  in  the  results.  In  addition,  the 
action  agency  may  provide  a  technical  report  for  those  projects  of 
generic  application  or  of  high  technology  interest.  IBEA  screens 
these  technical  reports  and  selects  those  suitable  for  publication  by 
the  National  Technical  Information  Service.  At  this  point,  the 
responsible  SUBMACOM  takes  the  necessary  action  to  implement  project 
results. 

5.3.  Proposed  Evaluation  Parameters. 

All  MANTECH  projects  can  be  evaluated  by  parameters  that  fall  into 
three  categories:  need,  risk,  and  economic  benefit.  These 

parameters,  illustrated  in  Figure  5-2,  can  be  combined,  as  discussed 
in  Section  5.5.,  to  determined  the  priority  or  rank  of  the  project. 

5.3.1.  Need.  All  MANTECH  projects  should  be  designed  to  satisfy  a 
documented  Army  need  to  improve  a  particular  manufacturing  process, 
technique,  or  equipment  in  a  manner  that  has  broad  applicability  not 
only  within  the  Army  but  also  within  other  branches  of  the  Armed 
Forces,  as  well  as  in  private  industry.  This  need  is  derived  from 
experience  with,  or  analysis  of,  the  ongoing  manufacture  and 
maintenance  of  existing  or  modified  military  systems,  as  well  as  the 
requirements  imposed  by  new  military  systems  entering  the  acquisition 
cycle. 
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Need  is  the  most  important  parameter  in  evaluating  TACOM  MANTECH 
projects  because  it  focuses  on  solving  existing  problems  or  providing 
solutions  for  potential  problems.  Existing  manufacturing  bottlenecks, 
new  system  designs  (including  material  changes),  new  system 
performance  requirements,  mission  support  requirements  (readiness, 
mobilization),  material  shortages,  and  maintenance  problems  contribute 
to  the  creation  of  a  need  for  MANTECH  projects.  The  criterion  of 
economics  also  can  be  expressed  as  a  need  (e.g.,  the  need  to  reduce 
costs),  although  it  is  more  generally  used  as  a  measure  of  merit  in 
evaluating  competing  projects.  Typical  candidate  manufacturing  areas 
that  generate  a  need  for  the  application  of  MANTECH  resources  include: 

t  Labor-intensive  operations; 

•  Operations  that  generate  high  waste,  particularly  of  scarce  and 
high-cost  material; 

•  Complex  operations  that  can  be  consolidated,  eliminated,  or 
streamlined; 

•  Operations  with  low  production  yields,  where  improvements  will 
reduce  rejection  rates; 

•  Dangerous  manufacturing  operations  (e.g.,  toxic  materials 
handling,  explosives,  pollutants,  and  equipment);  and 

•  General  high-cost  areas,  other  than  those  cited  above,  where  it 
is  known  that  a  particular  manufacturing  phase  represents  such 
a  large  percentage  of  costs  that  any  improvement  (manage¬ 
ment-material-process)  will  have  a  payoff  in  dollars  saved. 

General  objectives  or  parameters  that  typify  need  are 

•  Increase  Productivity:  Increase  Production  Rate  or  Improve 
Quality  (reliability,  availability,  maintainability,  and 
durability  (RAM-0) ) , 

•  Conserve  Critical/Costly  Material, 

•  Improve  Safety, 

•  Conserve  Energy, 

•  Reduce  Environmental  Impact,  and 

•  Reduce  Cost. 

Definitions  for  these  parameters  are  contained  in  Appendix  B. 
Reducing  costs  is  included  because  it  may  drive  a  specific  need. 
However,  reductions  in  cost  are  generally  realized  by  satisfying  one 
or  more  of  the  other  objectives.  In  these  instances,  cost  reductions 
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are  treated  separately  under  economic  benefit.  A  project  also  may 
have  secondary  benefits  by  satisfying  other  needs.  For  example,  a 
project  that  reduces  the  cost  of  curing  composite  materials  also  may 
improve  safety  or  reduce  environmental  impact.  In  these  cases,  extra 
credit  is  given  in  the  evaluation,  as  discussed  in  Section  5.5. 

The  establishment  of  needs  should  occur  on  an  annual  basis.  Although 
participation  in  this  effort  can  come  from  anywhere  within  TACOM,  or 
its  related  organizations,  it  is  essential  that  the  TACOM  MANTECH 
office  take  the  lead  in  establishing  the  list  for  it  to  serve  as 
effective  guidance  in  the  solicitation  of  MANTECH  proposals. 

Needs  should  be  succinctly  stated  and,  as  described  in  Section  5.2., 
should  be  based  on: 

•  Forcing  Function, 

•  Technology  Area, 

§  Subsystem  or  Component  of  the  Commodity,  and 

•  Commodity. 

Examples  of  such  statements  of  need  include 

•  Improve  safety  (forcing  function)  in  applying  caustic 
anticorrosion  paint  (technology  area)  to  all  tracked  combat 
vehicles  (commodity), 

•  Improve  the  quality  (forcing  function)  of  manufacturing 
(technology  area)  track  shoe  pins  (component)  for  Ml  tanks 
(commodity),  and 

t  Reduce  the  cost  (forcing  function)  of  curing  composite 
materials  (technology  area)  for  turret  baskets  (subsystem  or 
component)  of  the  Abrams  tank  (commodity). 

Once  the  total  list  of  needs  is  established,  it  is  necessary  to 
prioritize  them.  Again,  upper  management  participation  is  recommended 
because  (1)  it  may  be  necessary  to  cross  lines  of  equal  authority  to 
establish  priorities  and  (2)  endorsement  of  the  list  is  beneficial. 
In  establishing  priorities,  consideration  should  be  given  to  the 
following  criteria. 

•  Urgency.  What  is  the  impact  on  the  TACOM  mission  if  this  need 
is  not  satisfied?  Something  that  could  jeopardize  readiness  or 
mobilization  requirements  would  be  more  important  than 
something  that  just  reduces  costs. 

•  Timeliness.  Where  in  the  acquisition  cycle  is  the  hardware  for 
which  the  improvement  need  has  been  established?  In  most 
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cases,  the  earlier  the  need  is  satisfied,  the  more  effective  it 
will  be.  However,  there  also  may  be  cases  where  the  need  is  so 
great  that  it  should  receive  a  very  high  priority  or  it  will 
not  be  satisfied  before  the  production  run  is  completed. 

•  Genericness.  Is  it  a  broad  need  that  applies  to  several 
commodity  items,  subsystems,  or  components,  as  opposed  to  a 
unique  need  where  the  proposed  solution  may  be  less  likely  to 
have  generic  application? 

•  Interest.  Are  there  other  branches  of  the  military  or  private 
industry  that  may  be  pursuing  projects  based  on  a  somewhat 
similar  need?  Would  joint  participation  or  cofunding  with 
another  branch  of  the  service  be  possible?  Is  it  likely  that 
the  need  would  be  satisfied  in  the  near  term  without  TACOM 
pursuing  it  aggressively? 

5.3.2.  Risk.  Parameters  used  to  quantify  risk  have  been  limited  to 
likelihood  of  success  and  likelihood  of  implementation.  These 
parameters  are  defined  in  Appendix  B.  All  MANTECH  projects  will 
encounter  some  degree  of  risk.  This  risk  is  an  estimate  of  the 
probability  that  a  given  project  may  not  achieve  its  stated  goals  or 
objectives,  from  project  initiation  through  actual  implementation  in 
full-scale  production.  In  essence,  it  is  nothing  more  than  a 
reflection  of  the  uncertainty  band  around  key  evaluation  parameters 
whether  they  are  technical  or  economic.  For  example,  projects  will  be 
submitted  in  response  to  the  factors  described  in  Section  5.3.1.  that 
typify  need.  However,  there  is  always  the  chance  that  the  project  may 
not  fully  satisfy  the  need,  or  that  its  technical  feasibility  will  not 
prove  worthy  of  incorporation  into  full-scale  production.  Also,  even 
if  technical  feasibility  is  proven,  there  is  always  some  risk  that  the 
improvement  may  not  be  implemented  for  reasons  that  may  go  beyond  the 
control  of  the  MANTECH  organization.  Therefore,  there  is  a  definite 
uncertainty  associated  with  success  and  implementation. 

Economic  cost  and  benefit  estimates  also  are  likely  to  contain  some 
risk,  as  financial  estimates  are  nothing  more  than  projections  at  the 
early  stage  of  project  submittal.  However,  this  weakness  will  be 
common  to  all  projects  submitted.  Therefore,  relative  comparisons  of 
projects  will  be  consistent,  and  there  is  no  reason  to  become  overly 
sophisticated  in  attempting  to  bound  financial  figures  that  are  known 
to  be  estimates.  Furthermore,  for  multiyear  projects,  this  risk  is 
reduced  with  time  as  new  economic  analysis  data  are  produced  each  year. 

There  is  a  positive  correlation  between  the  technical  risk  of  success 
and  economic  risk.  The  greater  the  technical  risk,  the  greater  the 
likelihood  that  (1)  it  may  cost  more  than  projected,  even  if 
successful,  and  (2)  that  funding  may  be  invested  continually  in  the 
project  in  pursuing  modifications  that  may  make  it  successful,  when  in 
reality,  it  should  perhaps  be  dropped. 


22 


5.3.3.  Economic  Benefits  and  Costs. 

5. 3. 3.1.  Benefits.  For  the  case  of  TACOM  MANTECH  projects,  the 
economic  benefit  is  purely  one  of  cost  savings.  The  manufacturing 
work  must  go  on,  and  there  are  no  alternative  investment  scenarios  or 
profit  motive  incentives  that  apply.  Therefore,  production  cost 
savings  become  the  bottom  line;  it  is  important  to  understand  the 
traceability  from  need  to  the  dollars  that  are  saved.  Figure  5-3  is  a 
simplified  flow  that  illustrates  this  traceability;  it  shows  that  most 
economic  benefits  can  be  quantified.  It  is  essentially  a  cause  and 
effect  relationship  for  MANTECH  projects. 

As  illustrated  in  Figure  5-3,  all  MANTECH  projects  are  originated  to 
improve  the  manner  in  which  an  existing  commodity  is  being 
manufactured  or  to  arrive  at  a  method  for  manufacturing  some  new 
commodity  design.  These  two  objectives  also  are  interrelated  in  that 
the  manufacture  of  an  existing  commodity  may  be  altered  by  just  a 
change  in  product  materials,  as  opposed  to  an  entire  new  product 
design. 

These  MANTECH  requirements  can  basically  be  satisfied  by  new  or 
modified  (including  improved  quality)  equipment,  improved  factory 
flow,  or  changes  in  test  and  inspection.  These  changes,  in  a  sense, 
then  become  causes  that  either  taken  singularly  or  in  combination  will 
have  an  effect  on  the  current  method  of  production.  These  effects 
usually  appear  in  the  substitution  of  labor  or  in  general,  increased 
productivity  due  to  operating  more  efficiently,  reducing  the  downtime 
or  increasing  the  yield.  There  is  an  interrlationship  between  these 
effects  as  any  one  of  them  might  also  contribute  to  another,  e.g., 
reduced  downtime  will  certainly  increase  yield.  It  also  should  be 
noted  that  the  causes  are  all  branched  together  leading  to  the  effects 
because  there  is  more  than  a  one-to-one  correlation  between  cause  and 
effect.  For  example,  improved  quality  of  manufacturing  equipment  can 
reduce  downtime  and,  in  turn,  increase  yield. 

Each  of  the  effects  can  be  quantified  in  measurable  changes  in 
manpower,  overhead,  material  (kinds  and  amounts)  and  shortened 
schedules.  It  may  require  a  great  deal  of  research  to  derive  these 
quantities,  and  of  course,  the  precision  surrounding  the  answer  is 
directly  proportional  to  the  effort  put  into  obtaining  it.  The 
analyst  may  have  to  examine  the  Work  Breakdown  Structure  (WBS)  for  the 
item  being  produced  to  determine  which  specific  element(s)  are 
affected  and  to  what  degree.  Again,  there  is  an  interrelationship 
between  the  quantified  answers;  e.g.,  a  reduction  in  man-hours 
required  may  result  in  a  reduction  in  total  manpower,  which  should 
reduce  overhead.  For  most  TACOM  projects,  this  traceability  should  be 
quite  straightforward  and  values  should  be  derived  easily  because  they 
are  first-order  effects,  e.g.,  the  amount  of  time  saved  by  an 
individual  no  longer  required  due  to  automation  is  a  relatively  easy 
number  to  determine.  Other  changes,  such  as  improved  health  and 
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safety,  are  second-order  effects  and,  although  they  may  be  traceable, 
are  difficult  to  quantify.  The  effects  of  fewer  accidents,  fewer  days 
lost  due  to  illness,  etc.,  may  be  inherently  obvious,  but  may  yield 
insignificant  answers  in  an  economic  analysis.  In  these  cases,  more 
emphasis  should  be  placed  on  demonstrating  an  ability  to  meet  a 
priority  need  at  low  risk  than  on  factors  of  second-order  influence  in 
the  evaluation. 

The  quantifiable  effects  can,  in  turn,  be  expressed  in  dollar  amounts, 
e.g.,  man-hours  times  labor  rate,  pounds  or  feet  of  material  times 
unit  costs.  The  aggregate  of  these  dollar  amounts  then  becomes  the 
production  cost  savings,  which  is  the  benefit  to  TACOM. 

5. 3. 3. 2.  Costs.  Assuming  one  has  pursued  the  traceability  of  the 
manufacturing  change  and  has  arrived  at  a  "production  cost  savings," 
the  only  remaining  element  that  has  to  be  considered  is  the  actual 
cost  of  the  change;  i.e.,  hardware  and  facility  procurement  and 
construction,  etc.  Cost  savings  is  a  benefit,  but  to  make  this  a 
"net"  benefit,  the  cost  of  implementation  must  be  considered.  Another 
way  of  saying  this  is  that  production  cost  savings  are  often  pretty 
much  "operational"  costs;  i.e.,  the  line  is  up  and  running,  and  the 
only  thing  compared  is  running  it  the  existing  way  versus  the  improved 
way.  This  type  of  analysis  tends  to  ignore  acquisition  and  start  up 
costs  and  perhaps  operation  and  maintenance. 

Although  cost  figures  may  seem  easily  obtained,  often  provided  by  a 
reputable  supplier  of  the  improved  equipment,  considerable  attention 
should  be  given  to  their  derivation.  Several  things  should  be 
considered.  First,  is  the  forecasted  operation  and  maintenance  for 
the  new  equipment  and  how  favorably  it  compares  with  existing 
equipment.  This  may  have  been  considered  in  the  cost  savings  analysis 
(e.g.,  higher  quality  equipment,  less  downtime,  greater  yield).  Given 
that  MANTECH  demonstrations  may  involve  only  a  1-  to  3-year  effort, 
not  much  opportunity  is  afforded  to  build  a  historical  data  base,  such 
as  exists  for  the  current  equipment;  therefore,  this  estimate  may  be 
crude.  Second,  what  is  the  likelihood  that  the  existing  equipment 
will  soon  have  to  be  replaced,  even  if  not  improved;  net  costs  may  be 
reduced  if  the  choice  also  must  consider  buying  another  of  the  old,  as 
opposed  to  obsoleting  the  old  and  buying  new.  Third,  and  most 
important,  the  cost  estimate  should  consider  actual  implementation  of 
the  improvement  in  long-term  mass  production,  as  opposed  to  just  the 
costs  of  conducting  the  MANTECH  project.  Pilot  demonstrations  do  not 
have  the  same  impact  as  full-scale  development  changes  in  the  main 
production  line;  this  relates  to  the  ease  of  implementation,  which  is 
further  discussed  in  B-2.2. 

5.4.  Proposed  Evaluation  Procedures. 

The  methods  used  for  evaluating  and  ranking  candidate  projects  must 
not  ignore  the  total  process  of  developing  and  implementing  MANTECH 
projects  (as  represented  in  Figure  5-1).  Consideration  should  be 
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given,  therefore,  to  the  manner  in  which  proposals  are  solicited  and 
to  the  screening  factors  to  be  applied  to  them  before  submission  for 
review,  as  well  as  to  the  actual  evaluation  and  ranking  procedures. 
The  purpose  of  considering  the  total  MANTECH  process  is  to  develop  a 
standard  operating  procedure  that  complies  with  the  MANTECH  mandate  at 
TACOM.  The  procedure  should  be  known  to  all  participants  in  order  to 
maintain  a  continuity  of  effort  and  an  accountability  such  that  each 
proposal  is  handled  equitably.  Standardization  is  of  importance 
because  of  the  diversity  of  candidate  projects  that  must  be 
evaluated.  In  addition,  the  procedure  must  be  flexible  to  acconmodate 
changes  that  may  occur  in  the  policies  and  guidelines  set  by  higher 
authority  to  give  overall  definition  of  the  MANTECH  Program. 

The  documentation  and  current  program  cycle  schedule  discussed  in 
Section  4.0.  also  is  subject  to  change.  For  example,  the  two  separate 
parts  of  the  P-16  form,  with  separate  review  cycles,  was  instituted 
for  the  first  time  in  1983.  The  MANTECH  organization  of  TACOM 
responsible  for  these  procedures  must  have  a  method  for  informing  all 
parties  of  changes  in  the  process  and  develop  guidelines  to  respond  to 
these  changes.  The  real  possibility  of  major  changes  in  the 
definition  and  implementation  of  MANTECH  efforts  underscores  the  need 
for  not  only  a  systematic  process  that  is  documented  for  all  to 
follow,  but  also  a  systematic  method  for  modifying  the  process  in  a 
timely  manner.  The  important  phases  of  this  process  are  discussed 
below. 


5.4.1.  Proposal  Solicitation.  The  objectives  of  the  proposal 
solicitation  phase  should  be  to  obtain  proposals  that  (1)  meet  a 
significant  need  of  TACOM,  (2)  involve  effort  clearly  within  the 
province  of  MANTECH  funding,  and  (3)  provide  the  information  necessary 
for  evaluation  without  ambiguity.  Meeting  these  objectives  reduces 
the  time  and  labor  required  for  review  and  evaluation.  One  way  of 
promoting  these  objectives  is  to  provide  clear  guidelines  and  support 
to  those  preparing  proposals  and  to  maintain  a  year-round  contact  and 
liaison  with  those  groups,  both  government  and  industry,  whose 
function  is  the  manufacture  (or  purchase,  if  MACI  remains  a  part  of 
MANTECH  at  TACOM)  of  TACOM  commodities.  The  explicit  manner  in  which 
this  support  and  contact  is  achieved  depends,  of  course,  on  the 
organization  and  the  way  of  doing  business  that  each  SUBMACOM  has 
developed.  There  are,  however,  elements  of  this  activity  that  are  of 
importance  to  any  group  that  expects  to  develop  and  promote  quality 
MANTECH  projects  that  also  are  responsive  to  the  SUBMACOM  mission. 
These  elements  include 

•  Definition  of  priority  needs, 

t  Definition  of  screening  criteria, 

•  Definition  of  proposal  format,  and 

•  Sol itications  of  new  and  ongoing  project. 
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5. 4. 1.1.  Definition  of  Priority  Needs.  It  is  very  important  for 
TACOM  to  develop  a  priority  listing  of  MANTECH  needs.  Without  such  a 
listing,  the  evaluation  and  ranking  of  candidate  projects  tends  to 
become  more  subjective;  thus,  the  program  can  be  less  responsive,  more 
fragmented,  and  the  results  less  utilized  or  implemented.  As 
discussed  in  Section  5.3.,  need  is  identified  as  the  most  important 
evaluation  parameter.  The  selection  and  ranking  of  TACOM  needs  should 
receive  the  same  emphasis  and  support  for  MANTECH  as  does  the 
acquisition  process  for  major  new  systems. 

The  MANTECH  office  of  TACOM  should  take  the  lead  in  developing  a 
prioritized  list  of  needs  for  the  MANTECH  Program.  This  lead  may  be 
developed  through  discussions  with  those  directly  involved  or 
responsible  for  the  manufacture,  maintenance,  or  purchase  of  TACOM 
commodities.  The  prioritized  list  of  needs  should  be  updated 
annually.  The  definition  of  needs  should  strike  a  balance  between  the 
specific  and  the  general,  as  noted  in  Section  5.3.1.  Section  5.3.1. 
also  suggests  a  procedure  for  developing  the  priority  of  needs  once 
they  have  been  identified. 

5. 4. 1.2.  Definition  of  Screening  Criteria.  A  clear  statement  of  the 
type  and  scope  of  effort  that  bounds  the  MANTECH  Program  must  be 
provided  to  the  individuals  who  conceive  and  develop  MANTECH 
projects.  In  view  of  the  considerable  variety  of  interpretation  of 
what  constitutes  MANTECH  observed  during  the  survey  of  commands,  the 
development  of  screening  criteria  compatible  with  the  mission  of  TACOM 
is  of  great  importance. 

These  criteria  do  not  define  the  utility  of  a  candidate  project  but 
are  used  to  reject  projects  that  are  outside  the  current  definition  of 
MANTECH.  While  they  are  not  sacrosanct,  these  criteria  are  very 
useful  in  aiding  the  efficiency  of  both  the  proposer  and  evaluator. 
Section  5.2.6.  provides  a  listing  of  typical  criteria  obtained  from 
the  survey  of  commands  and  the  open  literature,  which  may  or  may  not 
fit  the  needs  of  TACOM.  A  frequently  overlooked  criterion  is  that  the 
project  must  not  improve  or  affect  the  design  or  performance  of  an  end 
item,  but  only  the  manufacture  of  an  end  item. 

It  is  required  that  the  MANTECH  group  at  TACOM  develop  and  maintain  a 
set  of  criteria  that  fits  the  current  definition  of  MANTECH  while  also 
meeting  the  needs  of  TACOM.  Determining  the  current  definition  of 
MANTECH  may  be  difficult,  but  some  defensible  assumptions  must  be  made 
and  validated  as  required  by  the  local  chain  of  command.  These  issues 
and  decisions  must  be  identified  in  the  beginning  and  made  visible  to 
all  participants  in  the  MANTECH  process. 

5. 4. 1.3.  New  and  Ongoing  Project  Solicitations.  The  solicitation  of 
new  and  ongoing  projects  requires  different  emphasis.  New,  worthwhile 
ideas  are  difficult  to  find  and  need  effort  to  cultivate,  as  with  any 
other  rare  commodity.  In  recognition  of  this,  some  commands  (e.g.. 
Materials  Laboratory,  Air  Force  Wright  Aeronatical  Laboratories 


27 


(AFWAL))  pursue  the  identification  of  new  projects  on  a  year-round 
basis.  Other  commands  conduct  seminars  and  meetings  periodically  to 
present  ideas  and  discuss  problems.  Briefings  also  are  conducted  to 
inform  industry  of  command  needs  and  opportunities.  It  does  seem 
prudent  to  give  the  listing  of  TACOM  needs  a  wide  distribution. 

Continuing  projects  as  well  as  new  projects  are  evaluated  each  year. 
The  evaluation  of  these  projects  should  consider  the  technical  status 
and  phase  of  each  in  addition  to  other  parameters.  Because  neither 
Part  I  nor  Part  II  of  the  P-16  form  contains  an  entry  regarding 
project  status,  TACOM 's  request  for  these  forms  may  ask  for  an 
addendum  to  provide  these  data  if  it  is  determined  necessary. 
Separate  project  status  reports  are  currently  provided  to  the  MANTECH 
group  semiannually.  Of  interest  to  evaluators  are  (1)  whether  the 
milestones  have  been  maintained  and  deliverables  provided 
successfully,  (2)  the  technical  problems  and  their  prognosis,  (3)  the 
growth  rate  of  costs,  and  (4)  the  percentage  of  funding  expended.  A 
project  that  is  demonstrating  progress  in  meeting  objectives  on  time 
and  in  budget  would  normally  receive  extra  credit  over  the  claims  made 
in  the  proposal  for  a  new  project.  The  evaluators  need  knowledge  of 
status  to  properly  rank  ongoing  projects  with  new  projects,  and  such 
information  should  be  requested  and  made  available  to  evaluators. 

5.4.2.  Request  for  Proposals.  The  process  for  implementing  quality 
proposal  measures  is  initiated  with  a  memorandum  to  the  individuals  or 
groups  known  to  be  potential  sources  of  projects  and  in  this  respect 
is  the  same  as  any  other  RFP.  The  memorandum,  or  call  letter,  should 
contain: 

•  A  priority  listing  of  TACOM  needs; 

•  Screening  criteria  and  definitions  or  clarifications  of  the 
MANTECH  Program,  as  appropriate; 

•  A  copy  of  the  DARCOM  guidelines,  if  appropriate,  and  P-16  forms; 

•  TACOM  guidelines,  as  appropriate;  and 

•  Due  dates  and  points  of  contact  in  the  MANTECH  group. 

It  would  be  well  to  also  include  the  evaluation  parameters  to  be  used 
and  a  brief  summary  of  the  evaluation  and  ranking  process  to  be  used. 
This  memorandum  should  be  issued  soon  after  the  updated  priority 
listing  of  needs  has  been  distributed,  anticipating  the  budget  and 
appointment  review  cycle  of  Figure  4-1. 

The  MANTECH  office  should  review  each  proposal  as  it  is  submitted  to 
determine  (1)  clarity  and  completeness  of  the  data  provided,  (2) 
adherence  to  the  guidelines  established  previously  by  IBEA  and  TACOM, 
and  (3)  that  each  passes  the  screening  criteria  for  MANTECH  projects. 
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The  determination  of  eligibility  by  screening  criteria  may  be 
judgemental  in  some  cases.  In  these  cases,  the  review  should  go  in 
favor  of  the  proposal,  with  the  final  determination  made  in  the 
evaluation  described  below. 

5.4.3.  Review  and  Evaluation.  It  is  proposed  that  the  MANTECH  group 
at  TACOM  establish  a  review  group  to  aid  the  evaluation  and  ranking  of 
projects.  Although  the  conduct  of  the  group  may  be  informal,  the 
following  characteristics  of  its  operation  are  recommended  to  help 
approach  the  desirable  goal  of  all  juries,  i.e.,  objectivity  with 
equity.  In  this  instance,  selection  for  success  is  the  real 
objective,  but  elusive.  One  can  only  hope  that  with  proper  selection 
of  evaluators,  objective  and  equitable  procedures  will  insure  a 
reasonable  success  rate. 

5.4.3. 1.  Evaluator  Selection.  The  MANTECH  group  should  identify  the 
kinds  of  expertise  required  to  assess  proposals  submitted  in  response 
to  the  needs  list.  It  is  suggested  that  the  review  group  consist  of 
permanent  members  who  are  informed  generalists  with  mature  judgment. 
Ideally,  each  permanent  member  should  have  hands-on  expertise  in  some 
aspect  of  manufacturing  that  compliments  the  expertise  of  other 
permanent  members  and  should  be  familiar  with  a  variety  of 
manufacturing  processes  and  techniques,  particuarly  those  related  to 
the  priority  needs  of  TACOM.  In  addition,  ad  hoc  members  can  be 
called  in  as  necessary  to  augment  the  panel  expertise.  The  panel  size 
should  be  small  enough  to  be  efficient  and  large  enough  to  insure 
coverage  of  the  majority  of  technical  issues,  possibly  5  to  10  members. 

The  review  panel  Should  be  chaired  by  a  representative  of  the  TACOM 
MANTECH  office.  The  chair  is  responsible  for  all  activities  related 
to  evaluation,  including  recruiting  members,  setting  agenda  and 
schedule,  developing  procedural  rules  considered  necessary,  conducting 
review  panel  meetings,  and  recording  findings. 

5. 4. 3. 2.  Agenda  Preparation.  The  chair  determines  the  agenda  for 
each  meeting  and  distributes  it  sufficiently  in  advance  for  each 
evaluator  to  prepare  for  the  meeting.  The  agenda  of  the  first  meeting 
of  the  review  group  should  be  accompanied  by  a  complete  data  package 
containing  all  the  information  previously  supplied  to  the  proposers. 
(If  the  evaluators  are  identified  and  recruited  in  time,  they  could  be 
provided  information  copies  of  the  RFP  package.)  Of  particular 
importance  is,  of  course,  the  definition  of  the  evaluation  and  ranking 
techniques  to  be  used  and  a  copy  of  each  proposal  to  be  evaluated. 
The  latter  will  have  been  checked  for  gross  errors  or  omissions  prior 
to  distribution. 

A  major  function  of  the  agenda  is  to  establish  the  order  in  which  the 
projects  are  to  be  considered  initially.  Whatever  the  order  adopted, 
there  could  be  an  advantage  to  considering  the  group  of  ongoing  . 


projects  first  because  some  of  the  new  projects  may  be  closely  related 
to  or  duplicate  those  of  ongoing  projects.  It  also  would  be  useful  to 
group  projects  that  are  related  or  propose  to  satisfy  the  same  need. 

5. 4. 3. 3.  Conducting  Meetings.  The  chair  should  not  only  determine 

the  agenda,  but  also  must  exert  authority  to  insure  it  is  followed. 
At  this  time,  it  should  be  determined  that  all  understand  the 

methodology  to  be  followed  in  evaluation  and  ranking  (Section  5.5.). 

It  is  recommended  that  the  evaluation  and  ranking  of  projects  be 
conducted  as  follows. 

t  The  first  meeting  of  the  review  group  should  be  brief  and 
limited  to  a  review  of  the  data  package  (Section  5. 4. 3. 2.) 
distributed  previously.  To  increase  efficiency,  it  is 
suggested  that  the  MANTECH  office  group  select  proposals  prior 
to  this  meeting,  allowing  the  chair  to  assign  a  group  of 
projects  to  each  member  that  is  compatable  with  his  or  her 

expertise.  The  purpose  of  this  assignment  is  to  ascertain 

facts  about  each  project  that  are  pertinent  to  its  evaluation. 
The  facts  to  be  established  include  the  status  of  continuing 
projects,  TACOM  needs  the  project  is  to  meet,  technical 
feasibility  or  relation  of  the  technology  involved  to  the  state 
of  the  art,  validation  of  economic  factors,  if  screening 
factors  are  violated,  etc.  Each  member  can  present  these  facts 
at  the  next  meeting,  thereby  allowing  more  time  for  evaluation. 

t  Subsequent  meeting(s)  are  devoted  to  evaluation  and  ranking  of 
projects  using  the  algorithm  defined  in  Section  5.5.  Each 
expression  in  the  algorithm  should  be  scored  for  each  project 
in  turn,  as  opposed  to  completing  the  entire  algorithm  for  each 
project.  This  technique  improves  the  relativity  of  ranking  by 
forcing  the  evaluator  to  concentrate  on  a  single  category  of 
parameters  for  all  projects.  All  members  of  the  group 
participate  in  this  process,  based  on  each  member's 
interpretation  of  the  facts  presented  previously.  The  average 
values  are  then  incorporated  into  the  complete  algorithm  and  a 
final  score  is  calculated  for  each  project.  This  score 
provides  the  ranking  order  of  all  projects  with  the  highest 
priority  project  receiving  the  highest  score. 

5. 4. 3. 4.  Recording  Results.  A  permanent  record  should  be  kept  of  the 
results  of  the  evaluation.  In  addition  to  the  rank  and  score  of 
projects,  a  record  of  the  minutes  of  each  meeting  should  be 
maintained.  The  minutes  should  be  available  to  members  at  the  next 
meeting  for  confirmation,  comment,  and  use  as  a  mnemonic.  The  minutes 
are  useful  for  preserving  continuity  in  the  evaluation  process,  for 
accountability  of  results,  and  can  be  useful  in  improving  the  process. 
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The  number  of  ways  the  evaluation  parameters  can  be  manipulated  to 
achieve  numerical  figures  of  merit  suitable  for  ranking  is  enormous. 
It  must  be  remembered  that  the  relationships  between  the  evaluation 
parameters  and  the  weights,  or  numbers,  applied  to  them  are 
established  in  an  arbitrary  manner  and  are  limited  only  by  the 
imagination  and  preferences  of  the  originator.  The  only  rule  is  the 
logic  that  a  positive  feature  of  a  project  increases  its  rank  and  a 
negative  feature  reduces  its  rank.  Therefore,  the  scoring  algorithm 
defined  herein  is  considered  to  be  merely  a  tool  to  simplify  the  task 
of  determining  an  initial  rank  order  of  a  large  number  of  candidates 
(approximately  30  to  40).  It  allows  the  consideration  of  features  one 
at  a  time  instead  of  all  at  once.  The  imposition  of  limits  on  the 
values,  or  weights,  applied  to  parameters  provides  some  objectivity  to 
the  ranking  because  the  values  are  applied  equally  to  all  projects. 
The  subjectivity  of  judgment  enters  in  making  the  estimate  of  the 
value  of  a  parameter  within  these  limits. 

The  algorithm  proposed  for  prioritizing  TACOM  MANTECH  projects  is 
based  on  combining  the  three  major  categories  of  need,  risk,  and 
economic  benefit,  taking  into  account  the  individual  parameters  that 
make  up  these  categories.  In  addition,  a  "status  factor"  was 
incorporated  into  the  algorithm  to  better  account  for  the  known 
performance  of  ongoing  projects. 

The  interpretation  of  the  scores  obtained  with  the  proposed  algorithm 
is  only  that  of  rank  order;  the  differences  between  scores,  or  actual 
values  obtained,  are  usually  not  important.  Therefore,  the  arithmetic 
required  to  calculate  the  score  has  been  kept  simple.  The  proposed 
algorithm  recommended  for  ranking,  or  prioritizing  TACOM  MANTECH 
projects  is: 

Economic 

Need  Risk  Benefit  Status 
100  (Rp  +  Bn)j  X  |^P5  X  Pjj  X  jsij  x 

The  expressions  in  the  algorithm,  and  the  suggested  values  or  limits 
to  be  applied  to  each  of  the  terms,  are  developed  in  the  following 
sections.  Table  5-1  summarizes  these  values  and  the  limits  that  are 
applied  to  each  term  of  the  algorithm.  The  ratios  of  maximum  to 
minimum  values  are  also  shown.  Because  the  algorithm  is  a  product  of 
terms,  these  ratios  are  indicative  of  the  influence  each  has  in 
establishing  rank.  As  can  be  seen,  the  values  of  these  ratios  for  the 
category  of  need  (7.5  N),  risk  (16),  and  economic  analysis  (3.3)  are 
roughly  equivalent  in  magnitude,  such  that  no  single  category  always 
takes  precedence. 


Table  5-1.  Characteristics  of  Scoring  Method 


Range 

Parameter 

Symbol 

Algorithm 

Minimum 

Maximum 

Ratio 

Need 

Priority  of  Need 
Relevance  of 

n 

100/n 

100/N 

100 

N* 

Project  to  Need 

Rn 

0.2 

1.0 

Secondary  Benefit 
Degree  of  Meeting 

Bn 

' 

0 

0.5 

Needs 

(Rn  +  Bn) 

0.2 

1.5 

7.5 

Risk 

Likelihood  of 
Success 
Likelihood  of 

Ps 

0.2 

0.8 

4 

Implementation 

Pi 

0.2 

0.8 

4 

Economic  Benefit 

Si  10 

3.0 

10 

3.3 

\l  1  /  \  /max 

1 

Status 

s 

(ongoing  projects) 

0 

2 

(new  projects) 

1 

1 

Score  =  10o|(Rn  + 

Bn)/"}  (Ps  • 

Pl)(Si)(S) 

0 

1920 

*Total  number  of  priority  needs. 
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5.5.1.  Need  Parameters.  The  algorithm  developed  here  assumes  that  a 
ranking  of  the  needs,  or  forcing  functions,  for  TACOM  has  been 
established.  Let  the  ranking  order  of  TACOM  needs  be  numbered  1,  2, 
...,  N  in  order  of  priority,  with  1  indicating  the  highest  priority,  n 
indicating  the  priority  of  the  nth  need  in  the  list,  and  N  indicating 
the  lowest  priority.  If  it  is  determined  that  the  project  under 
evaluation  is  proposed  to  meet  a  need  of  priority  n,  its  importance 
can  be  represented  by  100/n  (to  eliminate  small  decimals).  This  value 
is  then  modified  by  two  factors;  viz.,  the  relevance  of  the  project  to 
meeting  the  need  (Rn)»  and  the  secondary  benefits  (Bp)  that  may 
accrue  in  meeting  the  need. 

Rp  is  a  measure  of  the  degree  to  which  the  project  meets  need  "n." 
It  is  conceivable  that  the  project  only  attacks  part  of  the  problem 
represented  by  n,  or  that  its  schedule  may  not  exactly  meet  the  time 
table  of  need  "n."  In  those  rare  cases  where  a  project  appears  to  aid 
in  meeting  more  than  one  primary  need,  it  can  be  scored  under  each 
need  met  and  the  results  added.  A  secondary  benefit.  Bn,  is  one  not 
included  in  the  TACOM  needs  list  or  one  to  which  the  project  was  not 
directed,  but  that  is  of  general  benefit  to  the  military  production 
base.  The  establishment  of  a  project  whose  justification  is  to 
satisfy  the  need  to  reduce  the  cost  of  curing  composites  also  may 
increase  safety  and  reduce  pollutants  of  the  process. 


Let  Rp  and  Bp  assume  the  values 

0.2  <  Rp  <  1.0 
0  <  Bp  <  0.5 

The  choice  of  these  interval  bounds  can  be  defended  as  follows.  The 
relevance  of  the  project  (Rp)  should  be  greater  than  10  percent,  or 
it  should  not  have  been  identified  with  n.  An  upper  bound  of  "1" 
indicates  that  the  project,  if  successful  and  implemented,  will 
eliminate  need  (n)  from  the  list.  If  this  success  and  implementation 
provides  a  secondary  benefit(s),  it  also  then  receives  a  credit  of  up 
to  0.5.  Because  these  are  secondary  benefits,  their  value  should  be 
less  than  that  of  meeting  the  prime  need,  as  measured  by  Rp. 
Obviously,  if  there  are  no  secondary  benefits.  Bp  becomes  zero. 
Finally,  the  values  allocated  to  the  project  for  relevance  and 
secondary  benefit(s)  are  added  and  multiplied  by  the  importance  factor 
{ 100/n)  to  give  its  score  for  the  evaluation  category  of  need.  Thus 

Need  =  100  (Rp  +  Bp)/n. 

5.5.2.  Risk  Parameters.  There  are  two  kinds  of  risk  involved.  The 
first,  techical  feasibility,  is  measured  by  the  likelihood  of  success 
(Pg)  in  establishing  a  viable,  new  or  improved  manufacturing 
technique,  process,  or  equipment.  The  second,  implementation,  can  be 
measured  by  the  likelihood  of  implementation  (Pj).  Although  the 
likelihood  of  implementation  also  depends,  to  some  extent,  on  the 
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likelihood  of  success  (discussed  in  Appendix  B),  there  are  other 
factors  that  can  determine  if  a  project  is  to  be  implemented,  even  if 
successful.  Let  Pg  and  Pj  assume  the  values 

0.2  <  (Ps,  Pi)  <  0.8 

on  the  grounds  that  projections  of  this  nature  cannot  be  certain,  good 
or  bad.  The  likelihood  of  success  and  implementation  are  then 
multiplied  to  obtain  an  estimate  of  risk.  Thus 

Risk  =  Ps  *  Pi 

5.5.3.  Economic  Benefit  Parameters.  The  next  expression  in  the 
algorithm  is  one  that  determines  the  value,  or  weight,  of  economic 
rank.  As  developed  in  Appendix  C  and  Appendix  A,  the  benefit  to  cost 
ratio  (or  savings  to  investment  ratio)  was  selected  as  the  economic 
measure  of  merit.  However,  because  there  is  a  potentially  wide  range 
in  these  ratios,  it  was  determined  that  some  form  of  index  should  be 
developed  to  make  the  economic  component  of  the  ranking  algorithm 
consistent  in  scale  with  other  elements  of  the  algorithm.  This 
consistency  can  be  achieved  by  normalizing  the  values  of  the 
cumulative  benefit  to  cost  ratios  as  follows 

X  (Bi)/X  (Ci) 

10  J _ 1 

I  (Bi/CiOmax 

where  the  numerator  of  the  index  is  the  cumulative  benefit  to  cost 
ratio  summed  to  the  ith  project  of  the  set  being  evaluated  and  the 
denominator  is  the  maximum,  individual  benefit  to  cost  ratio  of  the 
set.  In  this  notation,  "i"  represents  the  economic  rank  of  a  given 
project,  with  (Bi/C-j)niax  representing  the  project  with  the 
highest  individual  B/C  ratio.  As  developed  in  Appendix  A,  the 
economic  rank  is  determined  by  a  computational  routine  that  compares 
projects  in  pairs  (recommended  if  computer  facilities  are  available) 
or  by  the  less  effective  method  of  comparing  individual  benefit  to 
cost  ratios. 

With  either  method,  the  use  of  incremental  cumulative  ratios  rather 
than  individual  benefit/cost  ratios  is  recommended  because  it 
represents  the  total  benefit  to  cost  ratio  that  can  be  realized  by  the 
subset  of  projects  to  be  funded.  It  also  provides  more  orderly 
increments  between  ranks.  (See  Table  A-1  of  Appendix  A). 

The  upper  bound  of  the  economic  index  is  10  because  it  is  normalized 
to  the  maximum  ratio.  (As  with  need,  the  multiplier  10  reduces  the 
prevalence  of  small  decimals.)  Theoretically,  the  lower  bound  of  Sj 
can  be  any  number  greater  than  zero.  Practically,  a  project  would 
have  a  ratio  greater  than  1  to  be  included  in  the  set  of  projects 
under  review.  The  use  of  cumulative  ratios  to  compute  Si  makes 
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index  values  approaching  1  very  unlikely.  The  minimum  value  of  S] 
shown  in  Table  5-1  was  that  obtained  for  the  sample  set  of  Appendix  A. 

5.5.4.  Status  Factor.  The  status  factor  (S)  is  designed  to 
accommodate  the  performance  history  of  ongoing  projects.  It  seems 
reasonable  to  propose  that  a  project  with  a  proven  history  of  good 
performance  should  receive  extra  credit  over  a  new  project  whose 
performance  can  only  be  promised.  Similarly,  a  project  with  poor 
performance  should  be  penalized.  The  score  of  a  new  project  should  be 
unaffected  by  S.  The  method  proposed  for  doing  this  is  to  have  S 
assume  the  following  values  as  a  multiplier  in  the  scoring  algorithm. 

0  ^  S  <  2,  for  ongoing  projects 
S  =  1,  for  new  projects. 

6.0.  CONCLUSIONS  AND  RECOMMENDATIONS 

6.1.  Conclusions. 


6.1.1.  Planning  Phase.  The  planning  process  for  Army  MANTECH 
projects  is  highly  structured  and  follows  procedures  and  requirements 
established  by  DOD  and  DMT.  The  three  principal  activities  of  the 
planning  phase  are  development  of  the  5-year  program  plan,  the  budget 
review,  and  the  apportionment  review.  Although  the  planning  process 
is  highly  structured  and  uniform  for  all  major  subordinate  commands, 
it  undergoes  frequent  review  and  is  modified  from  time  to  time  to 
improve  the  efficacy  of  the  process. 

6.1.2.  MANTECH  Program.  The  conduct  of  the  MANTECH  Program  is  a 
continuous,  iterative  process  in  which  the  review  and  evaluation  of 
proposed  projects  is  only  a  part.  The  principal  steps  are 

t  Monitor  and  evaluate  current  methods  of  commodity  manufacturing; 

•  Determine  priority  needs  for  improving  current  methods; 

•  Communicate  those  needs  to  all  parties  involved; 

•  Identify  candidate  projects  to  provide  the  priority  needs; 

•  Screen,  evaluate,  and  rank  projects; 

•  Select  and  conduct  projects  to  demonstrate  ability  to  meet 
needs; 

•  Disseminate  results  to  manufacturing  community  to  promote 
technology  development; 

f  Implement  proven  concept  in  mass  production;  and 

•  Assess  success  of  implementation  and  determine  new  priority 
needs. 


35 


The  only  unique  aspect  of  the  TACOM  MANTACH  Program,  as  with  any 
SUBMACOM,  is  its  product  line.  Therefore,  the  establishment  of  needs 
related  to  tactical  and  combat  vehicles  is  important. 

6.1.3.  Evaluation  Methods.  A  large  number  of  evaluation  parameters 
(~25)  were  obtained  from  a  review  of  regulations  and  from  a  survey  of 
DOD  commands.  It  was  concluded  that  there  is  a  great  degree  of 
commonality  among  many  of  these  parameters  because  they  are  different 
expressions  of  identical  concerns.  None  of  the  commands  contacted 
during  this  study  use  a  numerical  weighting  of  evaluation  parameters 
to  define  priorities. 

The  majority  of  commands  interviewed  have  developed  a  standard 
operating  procedure  (SOP)  or  internal  regulations  to  guide  the 
evaluation  process;  TACOM  does  not,  however,  use  an  SOP  or 
guidelines.  All  commands,  with  the  exception  of  TACOM,  also  use  a 
peer  review  group  to  select  and  rank  projects.  The  use  of  peer  group 
review  and  the  adherence  to  standard  procedures  cannot  by  themselves 
guarantee  objectivity  in  the  evaluation  process.  However,  they  are 
valuable  tools  that  provide  a  framework  for  a  more  objective 
evaluation,  and  the  TACOM  process  could  be  enhanced  if  it  employed 
these  features. 

All  commands  estimate  economic  benefit,  which  is  not  a  primary  factor 
in  ranking  projects. 

Because  the  majority  of  projects  are  multiyear,  the  evaluation 
methodology  must  consider  their  progress  and  status. 

The  evaluation  methodology  must  be  sufficiently  flexible  to 
accommodate  projects  that  vary  widely  in  technical  scope.  For 
example,  MACI  and  MMT  projects  are  different  project  classes. 

Improved  productivity  and  cost  savings  are  the  predominant  benefits 
associated  with  projects.  Almost  all  MANTECH  projects  provide 
improvements  that  can  be  related  to  cost  savings,  although  some  are 
second-order  effects  and  not  readily  defined.  A  review  of  the 
estimates  of  costs  and  benefits  contained  in  P-16  forms  submitted  by 
TACOM  reveal  a  wide  variation  in  the  depth  of  economic  analysis.  As 
expected,  estimates  depend  on  how  well  the  project  is  technically 
defined.  It  also  was  concluded  that  the  estimates  of  implementation 
costs  are  frequently  limited  to  conduct  of  the  MANTECH  project;  the 
true  cost  of  implementing  a  proven  MANTECH  concept  in  mass  production 
may  be  more  than  the  estimated  implementation  cost. 

6.2.  Recommendations. 

The  following  recommendations  are  presented  for  consideration  by  TACOM 
in  its  effort  to  improve  the  evaluation  and  ranking  of  MANTECH 
projects. 

6.2.1.  Prioritization  Methodology.  The  prioritization  methodology 
should  be  considered  as  part  of  a  continuing  MANTECH  process  and  not  a 
discrete  activity.  This  effort  should  include,  for  example,  year-long 
work  in  the  following  areas. 
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•  Identify  and  rank  priority  needs  in  the  manufacture  of  TACOM 
commodities  through  continual  liaison  and  personal  contact  with 
program  management  offices  and  others  directly  responsible  for 
commodity  manufacture.  Priority  lists  of  major  MANTECH  needs 
should  be  developed  and  maintained  by  the  TACOM  MANTECH  office. 

t  Communicate  those  needs  to  all  parties  to  promote  the  creation 
of  new  ideas  and  projects  that  may  meet  those  needs. 

•  Awareness  of  the  progress  achieved  and  problem  areas  of  all 
ongoing  MANTECH  projects. 

•  Promote  results  of  successful  projects  to  aid  the  likelihood  of 
implementation: 

—  Establish  and  maintain  user  contacts  to  cultivate  project 
sponsors, 

—  Keep  sponsors  informed  of  project  progress, 

—  Track  implementation  results  and  benefits,  and 
—  Evaluate  results  or  problems  to  identify  new  project  needs. 

Solicitation  and  development  of  new  ideas  should  be  considered  as  a 
year-round  activity,  including  discussion  with  all  parties  involved  in 
commodity  manufacture. 

6.2.2.  Standard  Operating  Procedure  and  Guideline.  An  SOP  and 
internal  guideline  for  developing  and  submitting  MANTECH  proposals 
should  be  developed.  In  particular,  TACOM  should  establish  a  review 
group  for  the  evaluation  and  ranking  of  projects.  The  selection  of 
its  members  and  the  procedures  it  follows  should  be  planned 
carefully.  An  SOP  would  provide  an  outline  for  the  review  and 
evaluation  process;  it  should  include  procedures  for: 

•  Screening  projects  and  rejecting  or  modifying,  as  required; 

§  Identifying  evaluator  requirements  and  selecting  and  procuring 
members  of  the  review  group. 

t  Appointing  a  chairperson  of  the  review  group  from  the  MANTECH 
organization  and  establishing  an  agenda. 

•  Obtaining  concensus  on  points  of  fact  regarding  each  proposal 
at  first  meeting  (subsequently,  scoring  each  proposal  using  the 
algorithm  below  and  ranking  projects  accordingly); 

•  Modifying  ranking  process,  if  required,  by  open  conment  and 
discussions;  and 

•  Maintaining  a  permanent  record  of  results  of  each  meeting. 
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A  set  of  guidelines,  to  be  included  with  a  formal  request  for 
proposals  that  defines  the  information  required  and  its  format, 
should  include 


f  Priority  listing  of  TACOM  needs; 

f  Screening  criteria  to  be  used,  with  clarifications  of  the 
MANTECH  Program,  as  appropriate; 

•  DARCOM  guidelines  and  P-16  forms; 

•  Internal  TACOM  guidelines,  as  appropriate;  and 

§  Evaluation  parameters  to  be  used  in  project  review. 


Guidelines  to  insure  consistency  in  the  development  of  estimates  of 
project  benefits  and  costs  should  be  developed.  For  example, 
consideration  should  be  given  to  the  following: 

•  Assume  10-year  production  runs  (or  procurement,  in  the  case  of 
MACI  projects)  beginning  1  year  after  completion  of  the  MANTECH 
demonstration; 

•  Establish  uniform,  yearly  inflation  and  discount  rates  to  be 
used  in  estimates  of  benefits  and  costs;  and 

t  Emphasize  the  need  to  estimate  and  include  production  line 
implementation  costs  (i.e.,  post-MANTECH) ,  including  operation 
and  maintenance. 


The  following  algorithm  should  be  used  to  compute  scores  to  obtain  an 
initial  ranking  of  projects. 


Score  = 

■lOO(Rn+Bn)' 

Ps  X  Pl 

Si 

s 

n 

where  n 


Bn 

Ps 

Pi 

Si 

s 


=  priority  of  need  met  (1.0  to  N) 

=  relevance  of  project  to  need  (0.2  to  1.0) 

=  secondary  benefit  (0  to  0.5) 

=  likelihood  of  success  (0.2  to  0.8) 

=  likelihood  of  implementation  (0.2  to  0.8) 

=  economic  index  (>0  to  10) 

=  project  status,  for  ongoing  projects  (0  to  2) 

for  new  projects  (1) 


The  economic  index  (Sj)  is  normalized  to  mitigate  the  large  excur¬ 
sions  observed  in  estimates  of  benefits  and  costs.  It  is  determined 
as  follows: 


Si  =  10 


L  (Bi)/  L  (Ct) 
1  1 
(Bi/^i )max 
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LIST  OF  ACRONYMS  AND  ABBREVIATIONS 


AFWAL 

AR 

A/R 

AMMRC 

D/R 

D/TR 

DA 

DESCOM 

DMT 

DOD 

HQDA 

HQ-DARCOM 

IBEA 

MAC  I 

MANTECH 

MMT 

MTAG 

NAC 

PBSP 

PEM 

PM 

RAM-D 

R&D 

RFP 

ROI 

SOP 

SUBMACOM 

TACOM 


Air  Force  Wright  Aeronatical  Laboratories 

Army  Regulation 

approval  pending  rewrite 

Army  Materials  and  Mechanics  Research  Center 

defer  for  rewrite 

defer  for  technical  reevaluation 

U.S.  Department  of  the  Army 

Depot  Systems  Command 

Directorate  of  Manufacturing  Technology 

U.S.  Department  of  Defense 

Headquarters,  Department  of  the  Army 

Headquarters,  U.S.  Army  Material  Development  and  Readiness 
Command 

Industrial  Base  Engineering  Activity 
Military  Adaptation  of  Commercial  Items 
Manufacturing  Technology 
Manufacturing  Methods  and  Technology 
Manufacturing  Technology  Advisory  Group 
Needs  Assessment  Committee 
Production  Base  Support  Program 
Production  Engineesring  Measures 
Project  Manager 

reliability,  availability,  maintainability,  durability 

research  and  development 

Request  for  Proposal 

return  on  investment 

Standard  Operating  Procedure 

Major  Subordinate  Command 

U.S.  Army  Tank-Automotive  Command 
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APPENDIX  A. 


RANKING  OF  CUMULATIVE  BENEFIT  TO  COST  RATIOS 


There  are  a  variety  of  classic  economic  methodologies  and  measures  of 
merit  suitable  for  project  prioritization.  The  benefit  (B)  to  cost 
(C)  ratio,  or  savings  to  investment  ratio,  was  selected  as  the 
preferred  measure  of  merit  for  MANTECH  projects.  This  ratio  is 
defined  as  the  net  economic  benefit  received  during  the  life  cycle  of 
the  new  process  or  technique  when  applied  in  mass  production  (or 
during  use  of  an  adopted  commercial  item)  divided  by  the  total  MANTECH 
project  cost  plus  its  implementation,  or  start-up,  cost.  In  computing 
this  ratio,  all  benefits  and  costs  are  converted  to  present  values  by 
use  of  inflation  and  discount  factors  as  stipulated  by  Army  Regula¬ 
tions. 

AR  11-28,  Economic  Analysis  and  Program  Evaluation  for  Resource 
Management,  provides  general  support  for  this  selection  and  guidance 
in  the  area  of  developing  an  incremental  merit  for  evaluating  and 
ranking  alternative  projects.  Specifically,  support  for  use  of  an 
incremental  approach  to  project  evaluation  is  found  in  Section  1-8(2), 
page  1-4  of  AR  11-28. 

...[The  purpose  of  the  economic  analysis  is 
to]  rank  these  alternatives,  based  on  their 
benefit  to  cost  relationship.  Economic  Analy¬ 
sis  is  designed  to  assist  the  manager  in 
identifying  the  most  cost  effective 
alternative  available  to  the  Army  for 
accomplishing  a  goal. 

With  respect  to  ranking  of  alternatives  and  selection  of  a  preferred 
alternative 

[In  situations  where  costs  and  benefits  of 
alternatives  are  unequal],  the  analysis  should 
compare  incremental  benefits  with  incremental 
costs.  Thus,  the  preferred  alternative  would 
be  the  one  with  the  largest  rate  of  effective¬ 
ness  to  cost.  [Section  2-3(f ) (3) (c) ,  pg.  2-6] 

AR  11-28  is  fairly  specific  in  terms  of  the  most  desirable  manner  in 
which  the  question  of  prioritization  should  be  addressed.  From  a 
purely  economic  perspective,  the  task  of  prioritizing  MANTECH  projects 
becomes  one  of  optimization  based  on  some  measure  of  cost 
effectiveness  per  dollar  spent. 

An  approach  that  can  be  applied  to  all  projects  for  which  an  economic 
analysis  has  been  accomplished  is  the  incremental  benefit/cost  ratio 
analysis,  whose  purpose  is  to  identify  the  most  cost-effective  means 
of  investing  limited  resources.  In  the  context  of  economic  analysis 
of  the  TACOM  MANTECH  Program,  projects  should  be  arrayed  to  reflect 
the  most  cost-effective  ordering  of  MANTECH  investment  alternatives. 
Alternative  approaches  were  explored  in  attempting  to  determine  a 
consistent  methodology  that  would  yield  such  an  array. 
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A  sample  of  MANTECH  projects  with  estimates  of  discounted  program 
benefits  and  costs  was  extracted  from  the  FY  1984  P-16  forms 
(Part  II).  This  sample  consisted  of  10  projects  of  varying  cost, 
ranging  from  approximately  $200,000  to  slightly  over  $2.8  million, 
with  individual  benefit/cost  ratios  ranging  from  49.52  to  1.18.  The 
objective  was  to  develop  a  consistent  approach  to  arraying  the 
projects  so  as  to  maximize  the  cost  effectiveness  per  dollar 
invested.  The  simplest  approach  is  to  rank  each  project  on  the  basis 
of  descending  benefit/cost  ratio,  invest  in  the  project  with  the 
highest  benefit/cost  ratio  first,  and  then  estimate  the  cumulative 
benefit/cost  ratio  by  adding  the  benefit  and  cost  of  each  successive 
project.  Let  i  represent  the  ith  project  in  a  set  of  projects.  The 
cumulative,  incremental  benefit  to  cost  ratio  (savings  to  investment 
ratio)  is  then  generated  for  the  ith  project  by 

r  (Bi)/  i  (Ci) 

1  1 

Table  A-1  presents  the  results  of  using  this  alternative. 

While  straightforward,  this  approach  will  not  consistently  yield  the 
highest  values  of  incrementation.  Theoretically,  it  is  possible  to 
enter  projects  in  order  of  descending  B/C  ratios  and  obtain  the 
highest  incremental  B/C  ratio.  However,  as  shown  below,  it  is  not  an 
optimal  approach;  an  alternative  approach  yields  higher  incremental 
B/C  values.  The  following  approach  was  taken  using  the  same  data  set. 

The  project  with  the  highest  individual  benefit/cost  ratio  is 
identified  first,  representing  the  most  effective  initial  investment 
when  funding  a  single  project.  The  project  of  second  rank  is  then 
determined  by  computing  (Bi+B-j  )/(C-|+Ci )  in  sequence  for  each 
of  the  remaining  (n-1)  projects  and  selecting  the  project  that  yields 
the  nighest  incremental  cumulative  benefit/cost  ratio. 

In  the  data  set  used  for  evaluation  purposes,  the  second  most 
incrementally  cost-effective  project  was  identified  as  project  No.  7, 
with  an  incremental  benefit/cost  ratio  of  45.35.  Adding  project  No.  7 
to  project  No.  1  results  in  the  greatest  addition  to  total  benefits  at 
the  lowest  cost.  Of  specific  interest  is  that  the  benefit/cost  ratio 
of  project  No.  7  is  2.87,  substantially  lower  than  the  second  highest 
overall  project  benefit/cost  ratio  in  the  data  sample  (i.e., 
82/02=13.09).  Similarly,  the  third-ranked  project  is  selected 
from  the  remaining  (n-2)  projects  by  choosing  the  project  that 
maximizes  (Bi+B;?+B-j  )/(C]+C7+C-j ) .  The  ranking  of  each 
remaining  project  is  determined  in  similar  fashion;  the  results  are 
listed  in  Table  A-2. 


Table  A-1.  Ranking  in  Order  of  Benefit-to-Cost  Ratio 


Project 

Number 

(ranking) 

(i) 

Benefit 

(Bi) 

Cost 

(Ci) 

Individual 

Benefit/Cost 

Ratio 

(Bi/Ci) 

Incremental 

Cumulative  Benefit/Cost 
Ratio* 

i  (Bi)/r  (Ci) 

1  1 

1 

101.02 

2.04 

49.52 

49.52 

2 

4.32 

0.33 

13.09 

44.45 

3 

1.84 

0.46 

4.05 

37.87 

4 

2.64 

0.66 

4.00 

31.47 

5 

3.68 

0.93 

3.97 

25.68 

6 

2.31 

0.70 

3.30 

22.62 

7 

0.56 

0.20 

2.87 

21.87 

8 

7.90 

2.82 

2.80 

15.27 

9 

0.54 

0.23 

2.33 

14.91 

10 

0.28 

0.23 

1.18 

14.55 

*Based  on  decreasing  value  of  individual  benefit/cost  ratio. 
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Table  A-2.  Comparison  of  Project  Rankings 


Incremental*  Incremental** 

Cumulative  Cumulative 

Project  Individual  Benefit/  Project  Individual  Benefit/ 

Number  Benefit/  Cost  Ratio  Number  Benefit/  Cost  Ratio 
(ranking)  Cost  Ratio  i  i  (ranking)  Cost  Ratio  i  i 

(i)  (Bi/Ci)  X(Bi)/X(Ci)  (i)  (Bi/Ci)  i:(Bi)/r(Ci) 

11  11 


1 

49.52 

49.52 

1 

49.52 

49.52 

2 

13.09 

44.45 

7 

2.87 

45.35 

3 

4.05 

37.87 

9 

2.33 

41.34 

4 

4.00 

31.47 

2 

13.09 

38.01 

5 

3.97 

25.68 

10 

1.18 

35.20 

6 

3.30 

22.62 

3 

4.05 

31.09 

7 

2.87 

21.87 

4 

4.00 

26.78 

8 

2.80 

15.27 

6 

3.30 

23.39 

9 

2.33 

14.91 

5 

3.97 

20.27 

10 

1.18 

14.55 

8 

2.80 

14.55 

*Based  on  decreasing  value  of  individual  benefit/cost  ratio. 

**Based  on  decreasing  value  of  the  ratio  of  cumulative  benefits  and  costs. 


Table  A-2  depicts  the  difference  in  project  rankings  resulting  from  an 
analysis  of  cumulative  incremental  benefit/cost  ratios  that  is  based 
on  descending  individual  project  B/C  ratios  versus  the  project 
rankings  that  result  from  use  of  cumulative  incremental  B/C  ratios. 
When  funds  are  constrained,  the  use  of  an  incremental  benefit/cost 
ratio  is  more  consistent  with  AR  11-28  in  terms  of  generating  the 
highest  economic  return  per  dollar  invested.  Theoretically,  a  better 
ranking  can  be  obtained  by  considering  all  possiole  combination  of 
projects,  and  computer  optimization  programs  are  available  to  do 
this.  However,  such  complexity  is  not  warranted  in  this  instance 
because  economic  benefit  is  not  the  only  evaluation  parameter  and  the 
technical  parameters  affect  the  overall  rank  of  each  project.  The 
simplified  optimization  scheme  demonstrates  that  substantial 
improvements  in  economic  benefit  can  be  identified  that  are  adequate 
for  assigning  the  weight  of  economic  benefit  to  project  rank.  It 
should  be  noted  that  this  simplified  approach  also  involves  a 
considerable  number  of  comparisons  to  be  made  when  the  number  of 
projects  approaches  40,  as  can  be  the  case  at  TACOM.  With  40 
projects,  approximately  700  comparions  are  involved.  However,  the 
Lotus  1-2-3  computer  program,  which  was  designed  for  this  kind  of 
calculation,  can  provide  a  spread  sheet  like  that  shown  in  Table  A-3. 

To  make  the  economic  component  of  the  ranking  algorithm  consistent  in 
scale  with  other  elements  of  the  algorithm,  it  is  recommended  that  the 
incremental  cumulative  benefit/cost  ratios  be  normalized  by  dividing 
them  by  the  maximum  individual  benefit-to-cost  ratio  of  the  set.  In 
addition,  as  shown  below,  the  multiplier  10  reduces  the  prevalence  of 
small  decimals. 


X  {Bi)/r  (Ci) 
Si  =  10  1  1 

(Bi/Ci )max 


The  calculation  of  the  economic 
the  indexes  listed  below. 

index  Sj  for  the 

sample  set 

Project 

Number 

Si 

Project 

Number 

Si 

1 

10.0 

3 

6.29 

7 

9.18 

4 

5.42 

9 

8.37 

6 

4.73 

2 

7.62 

5 

4.24 

10 

7.12 

8 

2.94 
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If  the  suitable  software  is  not  available  to  conduct  the  simplified 
optimization  method  recommended  herein,  the  economic  index  can  be 
determined  by  reverting  to  the  simpler  alternate  ranking  by  individual 
benefit/cost  ratio.  In  this  case,  the  sample  set  has  the  following 
indexes. 


Project 

Number 

Si 

Project 

Number 

Si 

1 

10.0 

6 

4.57 

2 

8.96 

7 

4.42 

3 

7.65 

8 

3.09 

4 

6.36 

9 

3.01 

5 

5.19 

10 

2.94 
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Table  A-3.  Cumulative  Incremental  Benefit/Cost  Ratio 


Project 

Number 

Project 
B/C  Ratio 

A 

B 

C 

D 

E 

F 

G 

H  I 

1 

49.52 

7 

2.88 

45.35 

9 

2.33 

44.74 

41.34 

2 

13.09 

44.45 

41.27 

38.01 

10 

1.18 

44.57 

41.26 

37.94 

35.20 

3 

4.05 

41.23 

38.44 

35.59 

33.31 

31.09 

4 

4.00 

38.41 

36.00 

33.52 

31.57 

29.65 

26.78 

6 

3.29 

37.67 

35.35 

32.96 

31.08 

29.22 

26.48 

23.39 

5 

3.97 

35.28 

33.27 

31.18 

29.57 

27.90 

25.44 

22.66 

20.27 

8 

2.80 

22.41 

21.65 

20.81 

20.36 

19.59 

18.47 

17.10 

15.84  14.55 

Column  Includes  Projects 


A 

B 

C 

D 

E 

F 

G 

H 

I 


1,  i 

1,  7,  i 

1.  7.  9,  i 

1,  7,  9,  2,  i 

1,  7,  9,  2,  10,  i 

1,  7,  9,  2,  10,  3,  i 

1,  7,  9,  2,  10,  3,  4,  i 

1,  7,  9,  2,  10,  3,  4,  6,  i 

1,  7,  9,  2,  10,  3,  4,  6,  5,  8 
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APPENDIX  B. 


GLOSSARY  OF  EVALUATION  PARAMETERS 


B-1.0.  NEED  CATEGORY 


For  the  need  category,  the  evaluation  parameters  are  synonymous  with 
the  parameters  that  typify  the  need  because  it  is  expected  that  the 
proposed  MANTECH  improvement  project  will  be  in  direct  response  to  the 
need. 


B-1.1.  Increased  Productivity. 

Increased  productivity  is  a  measurable  increase  in  the  number  of  units 
produced  in  a  given  time  without  increasing  production  capacity.  It 
can  be  achieved  by  means  that  directly  increase  the  rate  of  production 
or  by  improving  the  quality  of  production  that  indirectly  increases 
the  rate.  Advancement  in  the  state  of  the  art  is  subsumed  within  this 
parameter.  Such  increases  should  result  in  benefits  that  can  be 
quantified  in  the  economic  analysis. 

B-1.1.1.  Increased  Production  Rate.  An  increased  production  rate  is 
a  measurable  increase  in  the  number  of  units  produced  in  a  given  time 
by  improvements  in  processes  or  techniques,  by  increases  in 
efficiency,  or  through  equipment  changes  that  result  in  mechanization 
or  automation.  Although  increased  production  is  normally  a  laudible 
goal,  the  question,  "Why?"  has  to  be  asked.  In  general,  it  can  be  for 
one  of  two  reasons.  Either  the  current  rate  is  inadequate  to  meet 
requirements  reflected  in  the  current  Mission  Statement,  Army  Materiel 
Plan,  or  other  appropriate  Army  document,  or  costs  can  be  reduced  by 
producing  more  units  (if  required)  in  a  given  time  or  the  same  number 
of  units  in  less  time.  Which  of  these  is  important  should  be 
addressed  in  the  establishment  of  need. 

B-1.1. 2.  Improved  Quality.  Quality  is  the  umbrella  of  all  the  other 
"ilities."  It  is  difficult  to  envision  something  that  is  highly 
reliable,  very  durable,  and  easy  to  maintain,  and  yet  is  of  poor 
quality.  Also,  if  it  has  all  these  good  attributes,  it  most  certainly 
should  enhance  availability.  "Improved  quality"  is  often  cited  as  the 
justification  for  a  MANTECH  project;  however,  the  reference  is  usually 
to  the  quality  of  the  end  item  being  produced  and  not  the  quality  of 
the  manufacturing  change.  This  approach  is  not  entirely  valid,  and 
the  following  distinctions  should  be  made.  Quality  must  be  "designed 
into"  the  hardware,  whether  it  be  the  end  item  being  produced  or  the 
process,  technique,  or  equipment  that  produces  it.  There  is  no 
incentive,  or  real  payoff,  in  improving  quality  beyond  that  required 
by  the  design.  What  is  required  is  to  "insure"  that  the  designed 
quality  is  being  met  and  maintained. 

The  primary  focus  in  evaluating  MANTECH  projects  should  be  on  the 
quality  of  the  manufacturing  process,  technique,  or  equipment.  Test 
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and  inspection  of  this  improvement  change*  can  only  demonstrate  the 
degree  to  which  quality  is  being  maintained.  It  would  seem  quite 
natural  that  a  poor  quality  manufacturing  method  is  likely  to  produce 
a  poor  quality  end  item,  therefore,  in  the  event  one  starts  to  see  a 
degradation  in  the  quality  of  the  end  item,  it  may  be  wise  for  the 
hardware  design  engineer  to  get  together  with  the  production  design 
engineer  and  ascertain  the  true  cause.  This  degradation,  in  turn,  may 
dictate  a  new  MANTECH  need;  the  solution  for  which  will  again  be 
evaluated  according  to  the  other  "ilities"  mentioned  above. 

A  particular  MANTECH  project  under  evaluation  may  be  the  direct  result 
of  a  need  to  insure  quality  of  the  end  item.  More  sophisticated 
machinery,  tighter  controls,  and  increased  automation  are  examples  of 
changes  that  can  increase  precision  and  guarantee  reproducibility  or 
repeatability,  thereby  increasing  yield.  Quite  often,  however,  these 
projects  are  promoted  under  the  military  Quality  Assurance  Program, 
which  is  a  separate  DOD  activity. 

Improved  quality  of  manufacturing  will  result  in  less  downtime  or 
higher  yields,  benefits  that  can  be  quantified  in  the  economic 
analysis. 

B-1. 1.2.1.  Increased  Reliability.  Reliability  is  often  reflected  in 
terms  that  relate  to  the  performance  of  the  end  item  being  produced, 
and  not  the  manner  by  which  it  is  produced.  Producing  in  a  manner 
that  causes  poor  quality  or  high  rejection  rates  does  not  affect  the 
reliability  of  the  end  item,  which,  if  built  to  design,  would  have  the 
same  inherent  reliability;  it  just  makes  the  required  quality  more 
difficult,  time  consuming,  and  costly  to  achieve.  For  MANTECH 
projects,  however,  the  measure  of  reliability  should  be  restricted  to 
the  manufacturing  process,  technique,  or  equipment;  mechanisms  that 
are  not  large  repetitive  volume  buys  and  that  are  not  themselves 
subjected  to  lot  sampling  or  reliability  testing,  although  some  of  the 
piece  parts  that  go  into  these  mechanisms  may  be  so  tested. 

By  definition,  if  the  manufacturing  technology  project  is  to  be  an 
improvement,  it  cannot  degrade  reliability,  and  the  degree  of 
improvement  is  very  difficult  to  measure.  Therefore,  the  best  measure 
of  a  "more  reliable"  manufacturing  process,  technique,  or  equipment 
would  be  an  increase  in  production  rate  resulting  from  either  less 


*There  also  are  end  item  peculiar  "test  and  inspection"  MANTECH 
projects  that  should  not  be  confused  with  the  test  and  inspection  of 
the  hew  manufacturing  change.  MANTECH  test  and  inspection  projects 
represent  an  improvement  in  things  such  as  better  screening,  easier 
testing,  faster  screening  or  testing.  Some  of  these  can  directly 
enhance  the  quality  of  the  end  item  being  produced  by  better 
rejection  of  poorly  manufactured  components. 
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downtime  (as  mechanism  is  repaired)  or  lower  rejection  rates  (higher 
yield).  A  new  mechanism  that  just  produces  at  a  faster  rate  would  not 
represent  an  increase  in  reliability. 

B-1.1.2.2.  Maintainability.  By  definition,  any  improvement  in  a 
manufacturing  process,  technique,  or  piece  of  equipment  is  not  an 
improvement  if  it  degrades  maintainability.  Again,  one  has  to  be 
careful  to  distinguish  between  maintenance  of  the  production-oriented 
improvement,  as  opposed  to  maintenance  of  the  end  item  being 
produced.  For  MANTECH  projects,  the  emphasis  should  be  on  maintenance 
of  the  production  change.*  The  design  of  the  end  item  being  produced, 
as  opposed  to  the  manner  in  which  it  is  manufactured,  normally  affects 
the  maintenance  of  the  end  item. 

Measures  of  the  maintainability  of  the  production  change  are  reflected 
in  attributes  that  contribute  to  ease  of  repair,  extent  to  which  it 
must  be  replaced,  and  longevity  (mean  time  to  repair  or  replace). 
These  are  "life  cycle"  questions  that  apply  to  the  manufacturing 
change  and  not  the  end  product.  Any  possible  negative  impact  on 
maintainability  of  the  end  item  would  be  grounds  for  canceling  the 
project  and,  it  is  hoped,  these  relationships  would  have  been  defined 
in  the  R&D  or  feasibility  stage  prior  to  becoming  a  MANTECH  project. 

These  measures  should  be  reflected  in  the  economic  analysis  as  they 
are  related  directly  to  manufacturing  operation  and  maintenance  (O&M) 
costs  and  the  costs  of  varying  the  production  time  and  rate. 

B-1.1.2.3.  Availability.  As  with  maintainability,  an  improved 
manufacturing  process,  technique,  or  equipment  is  no  real  improvement 
if  it  is  not  available.  A  highly  precise  and  repeatable  piece  of 
machinery  that  is  unavailable  increases  downtime.  The  emphasis  for 
MANTECH  projects  should  be  on  the  availability  of  the  improvement, 
which  is  largely  a  function  of  its  reliability  and  maintainability,  or 
mean  time  between  failure  (MTBF)  and  mean  time  to  repair  (MTTR). 

Also,  a  case  where  there  just  "are  not  enough"  good  and  improved 
pieces  of  manufacturing  equipment  (i.e.,  unavailability)  is  an 
insufficient  reason  to  promote  it  as  a  MANTECH  project  and,  therefore, 
should  not  be  considered.  The  acquisition  of  additional  normal 
manufacturing  capability  to  expand  the  industrial  base  or  to  meet 
surge  and  mobilization  requirements  should  be  provided  by  normal 
production  appropriation  funds. 


♦There  are,  however,  some  MANTECH  projects  that  can  enhance 
maintenance  of  the  end  item.  For  example,  certain  improved  "test 
and  evaluation"  methods  or  equipment  may  be  better  able  to  predict 
failures;  thereby,  extending  the  useful  life  of  the  end  item  while 
allowing  for  corrective  action  before  the  failure  actually  takes 
place. 
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B-1.1.2.4.  Durability.  Again,  as  with  other  ilities,  one  should 
assess  the  durability  of  the  improved  manufacturing  process, 
technique,  or  equipment,  not  the  durability  of  the  end  item  being 
produced.*  "Will  the  new  mold,  die,  etc.,  hold  up  as  well  as  the  item 
it  is  replacing,  or  will  the  automated  handling  equipment  wear  out 
before  its  manual  counterpart?" 

B-1.2.  Conserve  Critical  and  Costly  Materials. 

The  need  to  conserve  critical  materials,  or  resources,  can,  by  itself, 
be  a  justification  for  a  MANTECH  project.  A  MANTECH  project  is  a 
solution  to  such  a  need  and  it  may  embody  the  substitution  of  new  or 
different  materials,  which  may  dictate  a  change  in  the  manufacturing 
process  or  just  a  change  in  the  manufacturing  process  that  eliminates 
waste.  In  certain  instances,  the  improvement  change  may  be  so 
demonstrative  (e.g.,  near  net  shape  versus  complex  machining  from 
scratch)  that  quantified  values  may  be  generated  for  the  economic 
analysis.  However,  it  may  quite  often  be  a  second-order  effect  and 
difficult  to  quantify. 

B-1.3.  Improve  Safety. 

Compliance  with  Occupational,  Safety,  and  Health  Administration  (OSHA) 
regulations  is  a  normal  TACOM  requirement,  and  manufacturing  changes 
to  accomplish  this  are  normally  funded  with  OMA  funds  or  PA  funds. 
However,  these  are  instances  where  the  application  of  MANTECH 
resources  would  be  appropriate.  For  example,  regulatory  compliance 
may  be  satisfied  through  the  use  of  protective  clothing  that  may  be 
awkward  or  bulky,  restricting  operations  or  limiting  the  amount  of 
time  that  the  worker  can  spend  on  the  associated  operation  without 
accepting  undue  risks.  A  manufacturing  improvement,  through 
automation,  may  actually  take  the  worker  out  of  the  loop,  thereby, 
improving  the  safety  of  the  operation  as  well  as  improving  the 
production  rate.  Although  a  case  can  be  made  that  this  improvement 
should  result  in  a  cost  savings  (e.g.,  fewer  lost  workdays,  more  units 
produced,  lower  overhead),  it  is  a  second-order  effect  and  difficult 
to  quantify  in  the  economic  analysis. 

B-1.4.  Conserve  Energy. 

The  need  to  conserve  energy  is  an  important  Army  program  given  special 
consideration  by  the  authority  of  the  "Army  Energy  Program,"  AR 
11-27.  Energy  savings  can  be  realized  through  new  breakthroughs  in 
manufacturing  improvement  that  would,  therefore,  make  them  candidates 
for  MANTECH  projects.  In  most  instances,  however,  energy  conservation 


*There  are,  however,  some  MANTECH  projects  that  can  enhance 
durability  of  the  end  item.  For  example,  a  change  to  a 
less-corrosive  material,  or  a  new  rubber  compound  that  makes  the  end 
item  more  durable,  may  require  a  new  manufacturing  method  to  produce 
it. 
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projects  are  of  such  a  general  nature  that  they  are  more  appropriately 
funded  with  OMA  funds. 

Given  other  higher  priority  needs  of  TACOM,  the  opportunity  for  a 
project  to  be  supported  solely  on  the  basis  of  energy  conservation  is 
quite  small.  However,  all  MANTECH  projects  present  the  opportunity 
for  conserving  energy  and  the  projected  savings  in  cost  (which  is  the 
best  measure  of  the  benefit,  as  opposed  to  Btu's  or  gallons  of  oil) 
can  readily  be  developed  and  should  be  incorporated  into  the  economic 
analysis. 

B-1.5.  Reduce  Environmental  Impact. 

Compliance  with  Environmental  Protection  Agency  regulations,  like 
safety,  is  a  normal  TACOM  requirement,  and  manufacturing  changes  to 
accomplish  this  are  normally  funded  with  OMA  or  program  funds.  Given 
other  higher  priority  needs  of  TACOM,  it  is  unlikely  that  a  MANTECH 
project  would  be  approved  solely  on  the  basis  of  this  parameter.  The 
benefits  of  controlling  pollution  are  hard  to  quantify  as  they  are  a 
function  of  the  type  of  pollutant,  the  geographic  area  of  the  source, 
the  amount  of  pollutant  being  discharged,  and  the  environmental 
pathways  for  its  impact.  Often  the  incorporation  of  pollution 
reduction  equipment  will  cost  more,  therefore,  a  savings  to  investment 
ratio  calculation  would  not  be  a  reason  to  promote  the  project. 

B-1.6.  Reduce  Costs. 

Cost  reduction,  a  somewhat  self-explanatory  parameter,  is  mentioned 
here  separately  as  it  may  be  synonomous  with  a  need  statement. 
Normally,  all  cost  reductions  will  fall  under  the  economic  analysis. 
The  challenge  to  reduce  costs  may  stem  from  a  rather  broad  need 
related  to  a  complete  commodity  item  (e.g.,  an  Industrial  Productivity 
Improvement  (IPI)),  where  many  opportunities  may  be  proposed,  to  a 
narrow  need  related  to  a  specific  operation  related  to  a  commodity 
subsystem  or  component. 

Cost  savings  can  result  from  changes  in  many  things,  such  as  facility 
layout  and  size;  modernized,  mechanized,  or  automated  equipment;  new 
process  or  technique;  improved  factory  flow;  personnel  reductions  and 
changes  in  skill  levels;  new  procedures;  reorganization;  and  reduced 
overhead. 

B-2.0.  RISK  CATEGORY 
B-2.1.  Likelihood  of  Success. 


Proposed  projects  should  present  practicable  alternatives  to 
manufacturing  processes,  techniques,  or  equipment.  The  proposed 
project  must  have  had  its  feasibility  demonstrated  (e.g.,  laboratory 
or  bench  scale)  prior  to  undertaking  it  as  a  MANTECH  project. 
However,  there  are  still  problems  that  may  be  encountered  as  scale-up 


B-6 


to  production  is  not  always  as  straightforward  as  one  would  like  it  to 
be.  Yet,  the  project  must  demonstrate  that  the  concept  is  suitable  to 
direct  transition  to  the  mass-production  manufacturing  floor. 

B-2.2.  Likelihood  of  Implementation. 

A  committed  sponsor  who  is  willing  to  implement  the  proposed  MANTECH 
project  is  a  very  important  asset  to  the  proposal.  There  can, 
however,  be  problems  with  implementation  even  if  the  PM  strongly 
supports  a  MANTECH  project  when  it  is  proposed.  After  the  project  is 
completed  and  ready  to  be  implemented,  the  management  of  the  program 
office  may  have  changed  and  there  is  a  chance  that  the  new  PM  may  not 
be  willing  to  accept  the  risk  associated  with  "doing  things 
differently  than  in  the  past."  This  situation  requires  placing 
continuing  emphasis  on  implementation  and  coordination  with  the 
program  office. 

There  also  is  the  chance  that  the  improvement  would  be  implemented 
without  the  benefit  of  MANTECH.  In  many  instances,  the  length  of  the 
budget  cycle  (2  to  3  years),  the  length  of  the  demonstration  project 
(up  to  3  years),  and  the  lead  time  to  get  the  proven  demonstration 
implemented  (1  year  estimated)  are  such  that  events  may  overtake  the 
idea  and  it  may  become  a  reality  without  MANTECH.  This  possibility 
raises  the  question,  "What  is  the  likelihood  of  the  project  being 
implemented  without  Army  sponsorship?"  A  question  appropriate  to 
initial  screening  of  MANTECH  proposals. 

Also,  consideration  should  be  given  to  the  "ease"  of  implementation. 
Would  implementing  the  project  cause  a  major  disruption  in  the  current 
manufacturing  process?  Will  the  production  line  have  to  be  shut  down, 
and,  if  so,  for  how  long?  Will  the  facility  require  major  structural 
modification  or  renovation?  How  many  procedures  or  shop  instructions 
must  be  modified? 
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INTERIM  PROGRESS  REPORT  FOR 
MANTECH  PRIORITIZATION  METHODOLOGY  PROGRAM 
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1.0  INTRODUCTION 


This  report  presents  the  results  obtained  to  date  on  the  Manufacturing 
Technology  (MANTECH)  Prioritization  Methodology  Program.  This  program 
consists  of  a  study  to  develop  a  method,  or  procedure,  for  evaluating 
and  selecting  candidate  MANTECH  projects  at  the  U.S.  Army 
Tank-Automotive  Command  (TACOM).  Selected  programs  are  subsequently 
submitted  to  the  Industrial  Base  Engineering  Activity  (IBEA),  Rock 
Island,  Illinois,  and  the  Materials  and  Mechanics  Research  Center 
(AMMRC),  Watertown,  Massachusetts,  for  review,  and  to  the  U.S.  Army 
Material  Development  and  Readiness  Command  (DARCOM)  for  approval. 
MANTECH  projects  to  be  evaluated  include  those  of  Manufacturing 
Methods  and  Technology  (MMT)  and  Military  Adaptation  of  Commercial 
Items  (MACI). 

Section  2.0  of  this  report  summarizes  the  work  of  Task  I  of  the  study, 
which  is  to  survey  the  Department  of  Defense  (DOD)  commands  involved 
in  the  evaluation  of  MANTECH  project  types,  to  review  their  procedures 
for  review  and  evaluation,  and  to  assess  the  applicability  of  these 
review  procedures  to  TACOM  needs. 

Section  3.0  presents  the  results  of  an  analysis  of  the  TACOM  FY  1985 
programs  submitted  to  IBEA  and  AMMRC  for  review,  and  their  ranking  by 
each.  An  analysis  of  the  review  comments  on  these  submittals  (i.e., 
P-16  forms)  given  by  IBEA  and  AMMRC  also  is  included. 

Section  4.0  addresses  the  accomplishments  of  the  work  on  Task  III  of 
the  study,  which  is  to  identify  the  manufacturing  and  operational 
parameters  to  be  used  for  the  evaluation  of  candidate  MANTECH 
proposals.  The  significant  parameters  are  selected,  defined,  and 
discussed  in  terms  of  utility  to  the  evaluation. 

Section  5.0  presents  a  discussion  of  methods  of  cost  analysis,  as 
required  by  paragraph  C.3.1.2  of  Task  III.  A  comparison  of  the 
classic  methods  is  given  that  identifies  the  savings-to-investment 
ratio  as  the  most  appropriate. 

The  work  remaining  to  complete  the  study  consists  of  the  completion  of 
Tasks  III  and  IV.  The  principal  subtask  to  be  completed  in  Task  III 
is  that  of  determining  the  method  of  relating  evaluation  parameters  to 
cost  factors.  In  Task  IV,  the  development  of  the  procedures  for 
engineering  review  and  analysis,  with  emphasis  on  the  organization  and 
operations  of  the  peer  review  group,  and  the  determination  of  the 
manner  in  which  risk  can  be  assessed  is  to  be  completed. 

2.0  REVIEW  OF  EXISTING  METHODOLOGIES 

A  survey  was  taken  of  the  procedures  used  by  other  military  commands 
to  evaluate,  rank,  and  select  MANTECH  projects  for  submittal  to  higher 
authority  for  approval.  The  purpose  of  the  survey  was  to  evaluate 
their  applicability  to  TACOM  for  the  evaluation  of  its  MANTECH 


C-2 


projects.  Points  of  contact  within  MANTECH  organizations  of  the  Army, 
Navy,  and  Air  Force  were  identified,  and  interviews  were  conducted. 
Figure  1  is  representative  of  the  steps  followed  by  the  major  system 
commands  in  processing  and  obtaining  approval  of  MANTECH  proposals. 
It  is  used  as  a  frame  of  reference  for  summarizing  the  findings  of  the 
survey. 

2.1  Army  Commands 

The  formal  process  of  initiating  MANTECH  projects  is  nearly  identical 
among  the  Army  commands  visited  because  they  all  must  follow 
requirements  instituted  by  the  Directorate  of  Manufacturing  Technology 
(DMT)  at  Headquarters  DARCOM,  as  promulgated  by  IBEA.  Differences  lie 
in  the  details  of  their  internal  review  methods,  the  evaluation 
parameters  selected  as  paramount,  and  the  screening  criteria  used. 
None  of  these  commands  used  a  numerical  algorithm  to  evaluate 
projects.  Although  a  few  had  supported  studies  to  devise  mathematical 
methods  for  decisionmaking,  none  were  ever  implemented.  A  significant 
number  follow  the  ranking  scheme  devised  by  AMRRC  (Ref.  1)  but  none 
use  it  in  arriving  at  their  final  internal  ranking  of  projects  (see 
Section  3.0) . 

2.1.1  Armament,  Munitions,  and  Chemical  Command  (AMCCOM),  Rock 
Island,  Illinois 


2. 1.1.1  Munitions 


2. 1.1. 1.1  Origin  of  Projects 

As  with  most  commands,  the  needs  perceived  by  the  engineers  and 
managers  are  the  prime  source  for  new  ideas  that  form  the  basis  for 
MANTECH  projects.  Those  closest  to  existing  manufacturing  processes, 
techniques,  and  equipment  also  are  closest  to  industry  counterparts 
and  are  most  familiar  with  the  state  of  the  art.  They  are  in  a 
logical  position  to  recommend  candidate  projects  by  virtue  of  their 
day-to-day  activities.  Although  ideas  for  improving  manufacturing 
techniques  can  and  do  occur  at  any  time,  their  transformation  to 
formal  proposals  follow  the  timely  process  of  responding  to  the 
January  call  for  proposals  issued  by  DARCOM,  and  responses  are  in 
accord  with  established  guidelines.  For  example,  the  Production  Base 
Modernization  Agency  in  Dover,  New  Jersey,  conducts  a  fall  guidance 
conference  for  the  Armament,  Munition  and  Chemical  Command  that  is 
followed  by  preliminary  submittals  to  working-level  meetings  before 
guidelines  are  formulated  for  the  preparation  of  the  P-16  forms. 

The  use  of  a  cost  driver  analysis,  or  "top  down"  study,  can  be  used  to 
originate  projects;  but  it  is  not  standard  procedure.  The  purpose  is 
to  determine  the  sources  of  the  largest  costs  in  the  production  of  an 
item,  or  group  of  items  in  a  given  facility.  Presumably,  the  sum  of 
these  large  cost  elements  for  a  production  activity  largely  determine, 
or  drive,  the  total  procurement  cost  and  offer  the  greatest  potential 
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Figure  1.  MANTECH  Project  Selection 


for  improvement  and  savings.  To  identify  the  large  cost  elements,  the 
production  activity  is  separated  into  major  categories,  such  as  parts 
or  material  processing,  fabrication,  assembly,  and  burden.  Each  of 
these  is  further  broken  down  into  subelements  as  fine  as  the  cost 
accounting  system  will  allow  for  the  allocation  of  costs  by  labor, 
materials,  equipment,  etc.;  hence,  the  term  "top  down"  study.  In  this 
manner,  large  cost  elements  are  highlighted  according  to  manufacturing 
activity,  each  of  which  can  then  be  analyzed  for  potential 
improvement.  Concern  was  expressed  that  emphasis  on  this  approach 
overstates  the  needs  of  a  single-weapon  system  at  the  expense  of 
projects  with  the  potential  for  generic  application. 

A  number  of  unsolicited  proposals  are  submitted  that  can  appear  at  any 
point  in  the  chain  of  command  where  the  contractor  perceives  a 
receptive  audience.  The  number  of  unsolicited  proposals  processed 
varies  between  subcommands,  but  is  probably  influenced  by  whether  they 
include  government  owned  government  operated  (6060),  government  owned 
contractor  operated  (60C0),  or  contractor  owned  contractor  operated 
(COCO)  facilities.  However,  they  are  handled  in  the  same  manner  as 
all  other  proposals.  There  is  always  the  critical  project,  mandated 
by  higher  command,  that  can  bypass  some  of  the  steps  of  Figure  1. 

In  summary,  projects  originate  at  the  working  level,  primarily  from 
engineers  active  in  manufacturing.  A  minor  portion  of  the  projects 
originate  as  unsolicited  proposals.  6uidelines  are  established  in  the 
fall  of  each  year  to  insure  that  candidate  projects  will  meet  the 
munition  needs  of  AMCCOM. 

2. 1.1. 1.2  Review  and  Evaluation 


Interdisciplinary  peer  review  is  the  dominant  method  used  for 
evaluating  and  selecting  projects.  This  is  true  not  only  for  AMCCOM 
but  also  for  every  command  visited.  Technical  experts  consisting  of 
project  engineers,  laboratory,  and  arsenal  personnel  are  selected  for 
the  review  panel.  Working-group  meetings  are  convened  to  select  and 
rank  projects  within  a  given  discipline  or  product  and  are  conducted 
informally  by  the  members  to  reach  a  consensus.  This  process  is 
followed  by  an  Executive  Review  Board,  which  evaluates  all  projects. 
The  Executive  Review  Board  meetings  include  representatives  of  IBEA 
and  AMMRC  for  coordination.  In  both  cases,  qualitative  methods  are 
used  to  make  decisions.  These  decisions  are  reached  by  considering 
the  following  factors: 

•  Screening  Criteria 

--  Satisfy  safety  and  pollution  control  regulations 
(Occupational  Safety  and  Health  Administration  (OSHA)  and 
Environmental  Protection  Agency  (EPA)) 

--  Return  on  investment  (ROI)  must  be  greater  than  1.5 

t  Evaluation  Parameters 

--  Projected  improvements  in  safety  and  pollution  abatement 
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“  Support  of  major  modification  and  expansion  programs 
--  For  new  programs,  satisfy  5-year  defense  plan  (FYDP) 

--  Support  mobilization  needs 
--  Uniqueness  of  technology  proposed 
--  Interest  of  command 

The  munitions  area  of  AMCCOM  is  unique  in  the  degree  of  concern 
expressed  for  safety  and  pollution.  Obviously,  considering  their 
product,  safety  is  important.  Also,  because  they  have  extensive 
manufacturing  responsibilities  as  a  single  weapons  manager  (i.e.,  sole 
source),  they  are  saddled  with  the  disposal  of  large  amounts  of  toxic 
waste,  including  NO^,  H2NO3,  heavy  metal  salts,  and  solid 

wastes.  Hence,  these  areas  receive  high  priority  in  the  evaluation. 

The  prioritization  process  is  qualitative,  and  no  numerical  scores  are 
used.  The  munitions  group  noted  the  difficulty  in  trying  to  make  the 
process  completely  objective.  However,  they  believe  their  method  of 
evaluation  is  quite  valid,  and  any  added  complexities  to  increase 
objectivity  are  not  worthwhile. 

2. 1.1. 1.3  Project  Award 

Once  the  P-16  proposal  passes  the  evaluation  of  the  Executive  Review 
Board,  is  ranked  high  enough  to  be  included  within  the  funding 

guidelines,  and  is  modified  as  required  for  the  DMT  review  at  DARCOM, 
preplanning  is  initiated  in  terms  of  the  preparations  of  the  statement 
of  work  structure  and  specification  of  deliverables.  The  goal  is  to 
hold  the  period  between  release  of  funds  and  initiation  of  contract  to 
about  6  months.  Once  a  project  is  under  way,  the  Production  Base 
Modernization  Agency  at  Dover,  New  Jersey,  holds  quarterly  reviews  on 
the  status  and  progress  achieved. 

2. 1.1. 2  Weapons 


The  weapons  group  at  AMCCOM  follows  the  procedure  described  above, 
except  for  the  following  characteristics. 

•  Screening  Criteria 

—  Improve  quality  assurance 

—  Satisfy  safety  and  pollution  control  regulations 
—  Be  aligned  with  the  Critical  Items  List  of  the  Army 

•  Evaluation  Parameters 

--  Support  mobilization  requirements 
—  Be  aligned  with  the  FYDP 
—  Have  the  greatest  economic  benefits 

In  comparison,  the  Weapons  Group  of  AMCCOM  used  a  simpler  selection 
process  in  terms  of  the  number  of  evaluation  parameters,  and  the  fact 
that  the  proposal  must  observe  the  Critical  Items  List.  The  Weapons 
Group,  perhaps  as  a  result  of  its  second  evaluation  parameter,  claimed 


C-6 


a  high  correlation  between  the  FYDP  and  projects  funded.  It  was  the 
only  command  interviewed  that  reported  a  high  correlation. 

2.1.2  Aviation  Systems  Command  (AVSCOM),  St.  Louis,  Missouri 

2. 1.2.1  Origin  of  Projects 

As  with  AMCCOM,  the  prime  source  of  projects  at  AVSCOM  are  the 
technical  staff  at  the  working  level.  Preparation  of  the  development 
of  ideas  and  preplanning  begins  in  October  in  anticipation  of  the  call 
letter  in  January.  AVSCOM  sends  call  letters  to  Langley  Air  Force 
Base  for  programs  related  to  structures,  to  Fort  Eustis  for  applied 
technology,  to  the  Propulsion  Laboratories  in  Cleveland,  and  to 
AMMRC.  It  was  noted  that  AVSCOM  considers  many  of  the  acronyms 
applied  to  production  engineering  programs,  such  as  technology 
modernization  (TECHMOD),  IPI,  IMIP,  to  be  basically  all  the  same  type 
of  work.  In  this  regard,  AR  700-90  and  AR  70-1  (Refs.  2  and  3)  are 
being  revised,  but  there  is  disagreement  on  how  it  should  be  modified. 

2. 1.2. 2  Review  and  Evaluation 

The  review  group  at  AVSCOM  consists  primarily  of  members  of  the 
Production  Technology  Branch  of  AVSCOM,  with  occasional  assistance 
from  specialists  from  the  laboratories.  There  is  no  formal  written 
procedure,  although  Battel le  Laboratories  conducted  a  study  (Ref.  4) 
for  AVSCOM  in  1976  on  the  subject  of  MANTECH  Program  guidance.  The 
main  emphasis  of  this  study  is  the  identification  of  promising 
projects  for  the  formulation  of  a  realistic  FYDP,  although  it  did 
include  recommendations  for  scoring  and  weighing  evaluation  parameters. 

•  Screening  Criteria 

--  Satisfy  regulatory  requirements  contained  in  AR  700-90 
(Ref.  2) 

•  Evaluation  Parameters 

—  Probabi 1 ity  of  implementation 
—  Savings  to  investment  ratio  (>3:1) 

--  Potential  for  generic  application 
--  Degree  of  technical  risk 

Much  of  the  review  panel's  work  centers  on  the  questions  of  proven 
feasibility  and  if  generic  application  can  be  achieved.  The  latter 
issue  is  resolved  by  determining  where  and  how  other  applications  of 
projects  could  be  obtained,  and  viewing  projects  pointed  to  a  specific 
weapons  system  (as  most  are)  as  a  demonstration  program  for  the 
concept  used  in  each  project.  In  general,  ongoing  programs  receive 
the  highest  priority. 

Concern  also  was  expressed  that  cost  savings  must  not  take  precedence 
over  technical  advances;  improvement  in  manufacturing  performance 
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should  be  of  equal  importance.  AVSCOM  also  confirmed  a  lack  of 
correlation  between  the  FYDP  and  the  projects  that  are  funded. 

2. 1.2. 3  Project  Award 

Most  of  the  AVSCOM  projects  that  are  funded  are  initiated  through 
competitve  procurements  with  industry  because  it  is  quicker  than 
sole-source  procurements.  Requests  for  Proposal  (RFP),  proposal 
evaluations,  and  contract  awards  are  done  at  the  laboratory  level 
where  the  work  will  be  conducted.  Each  project  also  is  monitored  by 
one  of  the  project  engineers  of  the  Production  Technology  Branch.  In 
addition  to  the  progress  reports  required  by  regulations,  in-house 
reviews  are  held  to  insure  progress  and  control  costs.  Although  the 
probability  of  implementing  projects  is  of  top  priority  during 
evaluation,  checks  on  the  implementation  of  successful  projects  are 
few.  Of  124  projects  conducted  between  1967  and  1979,  four  were 
tracked  to  calculate  savings  on  performance. 

2.1.3  Army  Missile  Command  (MICQM),  Huntsville,  Alabama 

2. 1.3.1  Origin  of  Projects 

Throughout  the  year,  conferences  are  held  in-house  and  with  other 
government  agencies  to  determine  perceived  needs  that  should  be 
supported.  This  process  is  a  principal  source  of  ideas  for  MANTECH 
projects.  Early  in  the  year,  a  call  letter  asking  for  projects  is 
sent  by  the  MMT  Division  to  all  MICOM  divisions.  The  Technical 
Integration  Office  develops  a  list  of  cormiand  priorities,  or  major 
thrust  areas,  that  is  provided  as  guidance  to  the  MMT  Division.  Thus, 
the  projects  that  are  contained  in  the  Part  I  P-16  form  represent 
these  command  priorities.  The  number  of  ideas  for  projects  that  are 
received  directly  from  industry  are  relatively  small.  The  major 
source  is  through  the  interaction  of  project  engineers  with  the 
relevant  industry. 

2. 1.3. 2  Review  and  Evaluation 


Expert  committees  are  formed  within  the  MMT  Division  to  review  these 
projects.  The  initial  selection  of  projects  for  the  Part  I  P-16  form 
is  a  screening  to  make  certain  that  each  meets  some  of  the  needs  of 
the  command  priorities.  There  is  no  numerical  ranking  done  at  this 
time;  rather,  these  Part  I  forms  are  sent  to  IBEA  for  review.  At  this 
point,  IBEA  may  question  the  feasibility  of  projects  but  normally 
reviews  them  from  the  standpoint  of  which  funds  are  used  in 
allocations,  and  checking  for  duplication  and  general  housekeeping. 
Following  this  review,  they  are  returned  to  MICOM  and  to  the  project 
engineer  who  originated  the  proposal.  Subsequently,  IBEA  travels  to 
MICOM  for  a  one-on-one  meeting  with  each  project  engineer.  It  is  here 
that  the  final  ranking  is  performed.  There  are  no  written  procedures 
on  the  ranking  process;  it  is  accomplished  through  the  use  of 
qualitative  considerations  rather  than  quantitative  factors. 
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•  Screening  Criteria 

--  Is  not  research  and  development  (R&D)  work 
—  Meets  a  command  need 

—  Is  a  clear  proposal,  i.e.,  follows  AR  700-90  (Ref.  2) 

—  Has  potential  for  generic  application 

t  Evaluation  Parameters 

--  Supports  end-item  producibility 
--  Reduces  production  lead  times 
--  Reduces  production  costs 
—  Increases  productivity 
—  Improves  process  reliability 
--  Conserves  energy  and  scarce  resources 
—  Insures  economic  availability  of  end  item 
—  Reduces  safety  hazards  and  pollution 

There  is  no  minimum  ROI  required,  although  this  analysis  is  done  by 
the  project  engineer  and  follows  AR  11-28A  (Ref.  5).  In  general, 
there  is  little  confidence  in  ROI  projections.  The  difficulty  with 
these  projections  is  that  the  amount  of  savings  accrued  depends  on  the 
number  of  people  who  will  pick  up  the  concept  and  use  it  in 
manufacturing,  which  is  almost  impossible  to  determine.  However,  a 
cost  driver  analysis  is  performed  for  all  MICOM  programs.  Critical 
needs  might  override  cost  considerations  and  could  result  in  the 
selection  of  a  project  that  has  a  negative  ROI. 

Auburn  University  conducted  a  study  (Ref.  6)  for  MICOM  on  the  use  of 
automated  selection  or  prioritization  of  projects.  The  final  report 
of  this  study  consists  of  a  complex,  theoretical  analysis  of  utility 
and  decisionmaking  theory,  and  includes  a  recipe  that  permits  one  to 
score  projects  according  to  the  degree  that  the  project  meets  six 
objectives  (see  Section  3.0).  This  method  has  not  been  implemented. 

The  MMT  Division  is  of  the  opinion  that  emphasis  should  be  given  to 
projects  that  are  applicable  to  the  concept  and  advanced  development 
phase  of  acquisition,  rather  than  the  engineering  development  or 
production  phase.  This  is  believed  necessary  to  support  production 
schedules  while  coping  with  the  time  it  takes  to  process  proposals  and 
get  projects  under  way,  which  is  difficult  because  regulations 
prohibit  research,  development,  testing,  and  evaluation  (RDT&E) 
funding  for  MANTECH  projects.  Another  difficulty  is  the  requirement 
to  evaluate  each  project  at  the  end  of  the  first  year  to  estimate 
benefits  to  cost  savings.  It  is  believed  a  longer  period  should  be 
used  for  this  evaluation. 

2. 1.3. 3  Project  Award 

All  projects  are  conducted  by  contractors.  These  procurements  may  be 
competitive  or  sole  source;  the  decision  is  left  to  the  discretion  of 
the  Procurement  Division.  The  MMT  of  MICOM  has  a  very  thorough  system 
of  promoting  and  tracking  implementation  results  after  project 
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completion.  A  member  of  the  MMT  staff  is  given  responsibility  for 
technology  transfer.  Video  tapes  of  the  processes  developed  by 
successful  projects  are  available  for  loan  to  help  sell  the  ideas 
(catalogues  of  these  are  sent  to  prospective  users)  and  seminars  and 
demonstrations  also  are  held.  Less  success  has  been  obtained  in 
estimating  savings  accrued  through  implementation.  In  many  cases,  the 
contractor  is  loath  to  provide  this  information  for  business  reasons. 
In  some  cases,  the  contractor  is  reluctant  to  admit  that  a  particular 
technique  has  been  incorporated  in  its  plant.  Nevertheless,  MICOM 
appears  to  have  an  excellent  program  for  evaluating  implementation 
effects,  and  has  managed  to  estimate  cost  savings  for  a  substantial 
number  of  projects. 

2.1.4  Industrial  Base  Engineering  Activity  (IBEA),  Rock  Island, 
Illinois 


2. 1.4.1  Origin  of  Projects 

IBEA  (and  AMMRC)  support  DMT  at  Headquarters  DARCOM  in  the  screening 
and  selection  of  projects  for  funding.  As  such,  they  are  not  involved 
directly  with  the  source  of  projects,  other  than  in  the  issuance  of 
call  letters  to  the  major  system  commands  in  January  of  each  year  and 
in  the  preparation  and  dissemination  of  the  FYDPs.  Their  principal 
role  is  to  insure  that  the  proposals  brought  forward  meet  MANTECH 
objectives  and  that  they  are  responsive  to  guidelines  established  by 
DARCOM. 

In  terms  of  the  phases  of  the  acquisition  cycle  (concept  phase, 
demonstration  and  validation  phase,  full-scale  engineering  development 
phase,  and  production  and  deployment  phase),  MANTECH  projects  appear 
to  occur  wherever  the  need  arises,  except  that  they  are  not  used  to 
produce  more  than  one-of-a-kind,  i.e.,  they  may  only  demonstrate  mass 
production  capability  as  a  pilot  run.  However,  IBEA  views  MANTECH 
projects  as  being  in  one  of  three  phases  designated  as  development, 
production,  or  maintenance,  with  the  development  phase  embodying  the 
first  three  phases  of  the  acquisition  cycle.  MACI  projects  are 
considered  part  of  the  late  development  and  early  production  phase, 
while  MMT  projects  are  initiated  primarily  in  the  production  phase. 

2. 1.4. 2  Review  and  Evaluation 


IBEA  reviews  all  projects  forwarded  by  the  major  systems  commands  in 
support  of  DMT.  IBEA  does  not  rank  projects  but  does  score  them 
according  to  the  following: 

f  Screening  Criteria 

—  Follow  guidelines  as  expressed  by  the  regulations  on 
submittal 
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•  Evaluation  Parameters 
--  Need 

Probability  of  success 
—  Potential  benefits 

The  scores  given  each  parameter  are  multiplied  together  to  obtain  a 
product  of  ratings  in  a  range  of  0  to  64.  AMMRC  also  scores  projects 
according  to  the  following  scheme,  based  on  the  MANTECH  objectives 
cited  in  AR  700-90  (Ref.  2): 


Evaluation  Parameters  Weight 


Pollution  abatement  10 
Production  safety  10 
Rate  of  ROI  8. 
Support  of  production  requirements  8 
Uniqueness  to  Department  of  Defense  (DOD)  production  5 
Critical  materials  savings  3 


The  scores  of  each  parameter  (from  -10  to  +10)  are  multiplied  by  the 
respective  weighting  factor  and  summed.  The  sum  is  then  multiplied 
with  the  scores  (1  to  3)  given  for  probability  of  success  and  for 
probability  of  implementation.  The  final  score  could  range  from  -1320 
(not  likely)  to  1320. 

After  reviewing  proposals,  IBEA  discusses  each  one  with  originators  at 
each  major  system  command,  recommending  changes  as  required  before 
submittal  to  DMT  for  final  review.  At  the  DMT  review  with  the  systems 
commands,  representatives  of  IBEA  (and  AMMRC)  are  present,  not  as 
advocates  but  as  consultants  and  coordinators. 

2. 1.4. 3  Project  Award 


After  projects  are  funded,  IBEA  supports  DMT  in  the  review  and 
tracking  of  project  performance  through  status  reports,  final  reports, 
and  effectiveness  reports  submitted  by  project  engineers.  IBEA 
summarizes  these  results  by  preparing  consolidated  reports  for  DMT 
review. 

2.2  Air  Force  Commands 


2.2.1  Air  Force  Wright  Aeronautical  Laboratories  (AFWAL),  Dayton,  Ohio 

The  AFWAL  MANTECH  Division,  Materials  Laboratory,  is  the  lead  agency 
of  the  Air  Force  for  MANTECH  and  is  analogous  to  the  Army  DMT.  AFWAL 
is  responsible  for  the  planning,  evaluation,  and  selection  of  projects 
to  be  implemented  by  the  Air  Force.  While  there  is  no  formal 
documentation  of  the  procedures  followed  by  the  MANTECH  Division,  they 
are  very  similar  to  those  of  the  Army. 
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2. 2. 1.1  Origin  of  Projects 


Projects  are  solicited  from  the  various  product  divisions  of  the  Air 
Force  by  the  MANTECH  Division.  The  product  divisions,  in  turn, 
routinely  solicit  industry  for  project  ideas  related  to  critical 
areas.  Industry  briefings  are  held  in  each  focal  point  area  to  inform 
them  of  needs  and  opportunities. 

2. 2. 1.2  Review  and  Evaluation 

When  received,  projects  are  collated  into  groups  of  similar 
objectives,  called  focal  areas  or  focal  point  efforts;  these 
correspond  to  critical  areas  of  technology  necessary  to  Air  Force 
systems  development.  Development  of  focal  point  efforts  is  a 
year-round  effort  by  the  MANTECH  Division,  with  inputs  from  the 
product  divisions,  and  corresponds  to  what  are  called  thrust  areas  in 
some  of  the  Army  commands.  Because  projects  are  processed  explicitly 
according  to  which  focal  point  they  support,  great  emphasis  is  given 
to  the  criteria  that  projects  meet  an  identified  need  of  the  Air 
Force.  Current  focal  points  include: 

•  Nonmetals 

•  Metals 

•  Electronics 

•  Thermal  Protection 

•  Nondestructive  Inspection 

•  Propulsion 

•  Computer-Aided  Manufacture  (CAM) 

An  Executive  Committee,  consisting  of  Division  Chiefs,  the  Chief  of 
Operations,  and  the  Chief  of  Plans  for  the  Materials  Laboratory, 
appoint  focal  point  leaders  who,  in  turn,  appoint  members  to  form  a 
peer  review  panel  to  review,  evaluate,  and  prioritize  candidate 
projects.  A  representative  of  the  MANTECH  Division  is  a  member  of 
each  focal  point  panel.  As  with  other  commands  contacted  in  this 
survey,  the  focal  point  panels  depend  on  qualitative,  rather  than 
quantitative,  factors  in  establishing  priorities.  Estimates  of  ROI 
are  largely  ignored  in  this  process.  Of  greater  importance  is  the 
likelihood  of  the  project  improving  system  performance,  or  the 
"ilities",  i.e.,  reliability,  availability,  maintainability,  durabi¬ 
lity  (RAM-0).  Screening  factors  related  to  safety,  energy,  or  pollu¬ 
tion  are  not  involved. 

The  findings  of  each  focal  point  panel  are  submitted  to  the  Executive 
Committee  for  review  and  then  to  the  MANTECH  Division,  which  prepares 
a  final  report  for  submittal  to  the  Air  Force  Systems  Command  (AFSC), 
Deputy  for  Contracting  and  Manufacturing.  In  preparing  this  report, 
which  includes  all  Air  Force  MANTECH  projects,  some  restructuring  of 
priorities  may  occur  in  order  to  meet  the  budget  limitations  of  each 
product  division.  In  this  sense,  projects  do  not  compete  across 
product  divisions  in  this  process,  although  this  may  occur  in  the 
focal  point  evaluation  process. 
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2. 2. 1.3  Project  Award 


The  procurement  cycle  is  approximately  6  months,  with  a  total  time  of 
approximately  8  months  between  proposal  submittal  and  start  of 
contract.  Approximately  80  percent  of  the  funded  projects  are 
solicited  from  industry;  only  5  percent  represent  unsolicited 
proposals.  Approximately  70  percent  represent  ongoing  efforts.  Work 
is  just  beginning  on  evaluation  of  the  success  rate,  implementation 
rate,  and  cost  benefits  of  funded  projects. 

2.2.2  Ballistic  Missile  Organization  (BMP),  San  Bernardino,  California 

BMO  is  not  involved  directly  in  MANTECH  projects  because  they  are 
handled  by  the  MANTECH  Division  at  AFWAL.  BMO  may  provide  project 
ideas  as  a  result  of  solicitation  from  the  MANTECH  Division  but  are 
not  otherwise  active  in  that  area.  The  work  at  BMO  is  concerned 
solely  with  TECHMOD.  The  objective  of  TECHMOD  is  to  "accelerate  the 
implementation  of  advanced  manufacturing  technology  through  the 
coupling  of  contractual  incentives  with  technology  development" 
(Ref.  7). 

2. 2. 2.1  Origin  of  Projects 

BMO  does  not  currently  have  any  TECHMOD  projects  but  is  awaiting 
proposals  from  three  contractors.  The  method  for  initiating  projects 
is  to  originate  plans  in-house,  brief  relevant  contractors  on  these 
plans,  and  solicit  proposals.  BMO  refers  to  three  phases  in  TECHMOD 
projects. 

Phase  I:  Top  down  studies  and  analyses  of  what  needs  to  be  done 

and  projected  benefits. 

Phase  II:  Prototype  demonstration  of  Phase  I  concepts  and 
compare  results  with  predictions. 

Phase  III;  Apply  successful  prototypes  to  a  manufacturing 
environment. 

It  is  planned  for  the  first  phase  to  be  funded  by  the  contractor,  with 
incentives  to  proceed  to  the  second  and  third  phases  provided  by  BMO. 

2. 2. 2. 2  Review  and  Evaluation 

BMO  will  rely  on  the  support  of  the  technical  expertise  resident  in 
the  various  product  managers'  offices  for  proposal  evaluations. 
Because  of  the  recent  birth  of  this  effort,  project  selection  is 
implicit  in  the  in-house  process  of  origination.  However,  the 
following  evaluation  parameters  are  used  in  the  approval  of  contractor 
efforts. 
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•  Support  a  specific  application  or  need 

•  Criticality  of  need  being  supported 

•  Reduce  costs 

•  Timeliness 

There  is  no  threshold  of  economic  benefits,  nor  are  there  guidelines 
on  how  to  estimate  benefits.  However,  they  must  be  reasonable  and 
verifiable. 

2. 2. 2. 3  Project  Award 


Competitive  proposals  are  considered  impractical.  The  reasoning  is 
that  one  contractor  will  not  accept  the  risk  of  an  approach  or 
application  proven  by  another  contractor  unless  funded  to  demonstrate 
it  again;  this  is  one  of  the  difficulties  in  achieving  technology 
transfer. 

2.2.3  Space  Division  (SD),  El  Segundo,  California 


2.2.3. 1  Origin  of  Projects 


A  call  letter  for  projects  is  issued  from  AFSC  to  program  offices  in 
the  first  quarter  of  each  calendar  year.  ^  The  program  office  develops 
projects  both  internally  and  from  inputs*  obtained  from  contractors, 
i.e.,  similar  to  the  methods  used  by  Army  commands.  Proposals  are 
prepared  using  a  simplified  version  of  the  P-16  form. 

2. 2. 3. 2  Review  and  Evaluation 


Projects  are  presented  to  a  senior  MANTECH  review  group  that 
subsequently  ranks  the  proposals  before  forwarding  them  to  AFSC  (i.e., 
AFWAL). 

•  Screening  Criterion 
—  No  R&D 

--  No  duplication  of  effort 
—  Generic 

•  Evaluation  Parameters 

—  Criticality  of  project 
--  Project  funding  level  required 

—  Projected  improvements  in  system  efficiency  and  life 

Each  project  is  rated  by  all  members  of  the  review  group,  and  a 
consensus  on  the  prioritization  of  projects  is  reached.  This  process 
is  informal,  and  no  quantitative  methods  are  used  in  deriving 
priorities.  There  are  no  written  guidelines  or  descriptions  of  these 
procedures.  Highly  detailed  economic  analyses  are  not  conducted. 
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2. 2. 3. 3  Project  Awards 


Most  MANTECH  SD  projects  last  2  to  3  years.  No  details  were  available 
on  whether  effectiveness  studies  of  these  projects  are  conducted  by  SD 
or  the  proportion  of  competitive  to  sole-source  awards. 

2.3  Naval  Material  Command  Industrial  Resources  Detachment  (NMCIRD), 
Philadelphia,  Pennsylvania 


NMCIRD  provides  technical  support  to  Naval  Material  (NAVMAT)  on 
MANTECH  projects  and  has  responsibilities  similar  to  those  of  IDEA  in 
support  of  DMT.  NMCIRD  is  organized  along  the  lines  of  the  MTAG 
subcommittees,  with  an  engineer  heading  each  of  the  six  disciplines. 
NMCIRD  may  request  assistance  from  technical  staff  members  of  other 
system  commands  when  required.  Because  the  first  MANTECH  projects 
were  begun  in  1977,  the  MANTECH  Program  at  NMCIRD  is  just  beginning  to 
mature. 

2.3.1  Origin  of  Projects 

The  prime  source  of  projects  are  those  offered  by  contractors  working 
on  Navy  programs.  In  the  future,  NMCIRD  plans  to  emphasize  larger 
programs  for  major  weapons  systems  through  the  use  of  "top  down" 
studies  for  identification  of  projects.  Cost  driver  analysis  is 
considered  a  valuable  asset  in  this  identification. 

2.3.2  Review  and  Evaluation 


NMCIRD  does  not  participate  in  the  evaluation  of  projects  prior  to 
receiving  them  for  review.  Each  command  has  received  budget 
appropriations  from  NMCIRD,  and  the  command  submittals  generally  match 
their  budgets.  Thus,  there  is  really  no  competition  among  the 
commands  for  project  funds.  NMCIRD  does  reject  proposals,  which  is 
based  solely  on  the  project's  lack  of  compliance  with  NAVMAT 
instructions  (NAVMATINST  4800. 36D)  (Ref.  8). 

The  screening  criteria  in  these  instructions  are: 

•  Satisfying  a  Navy  requirement  to  provide  increased  productivity 
or  process  applicability; 

•  Solving  problem  solutions  through  the  use  of  new,  improved,  or 
more  economical  manufacturing  processes,  methods,  techniques, 
or  equipment; 

•  Incorporating  state-of-the-art  capability; 

t  Not  duplicating  effort; 

•  Being  beyond  the  normal  risk  of  industry; 
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•  Having  a  reasonable  payback  time;  and 

•  No  R&D  effort. 

NMCIRD  does  not  use  a  numerical  ranking  scheme  in  assessing  its 
evaluation,  and  the  economic  assessment  is  somewhat  limited  in  that  it 
is  based  on  a  payback  time  of  5  years,  beginning  with  the  initial 
release  of  funds.  Given  that  most  projects  take  3  years  to  be 
demonstrated  plus  another  year  before  they  are  implemented,  one 
certainly  cannot  have  "life  cycle"  costs  taken  into  consideration  with 
only  1  year  of  production  considered. 

NMCIRD  does  not  consider  qualitative  or  subjective  benefits,  such  as 
RAM-D.  It  is  assumed  that  the  proposed  projects  do  not  violate  or 
adversely  impact  RAM-D;  the  extent  to  which  they  are  improved  is 
secondary  to  the  main  objective  of  the  program.  Likewise,  no  specific 
attention  is  paid  to  pollution  abatement,  safety,  or  energy 
conservation.  OSHA  and  ERA  regulations  govern  these  parameters  and 
they  are,  therefore,  not  of  themselves  justification. 

In  line  with  the  above,  NMCIRD  agrees  that  any  improvement  in 
reliability  should  be  restricted  to  the  manufacturing  process, 
technique,  or  equipment  and  not  to  an  improvement  in  the  reliability 
of  the  end  item.  The  latter  may  be  a  result  of  a  design  change  or  a 
change  in  requirements,  albeit  a  new  requirement  may  dictate  a  change 
in  manufacturing.  Therefore,  mean  time  between  failure  (MTBF)  or  mean 
time  to  repair  (MTTR)  are  not  worthy  of  consideration  unless  there  is 
some  way  to  prove  that  the  "manufacturing  process"  alone  contributed 
to  these  benefits. 

2.3.3  Project  Award 

Overall,  approximately  25  percent  of  NMCIRD  projects  are  implemented. 
2.4  Summary  of  Findings 

Results  of  the  survey  detailed  in  the  above  discussion  are  summarized 
in  Table  1.  The  following  characteristics,  which  typify  the 
processing  and  evaluation  of  MANTECH  projects  by  the  commands  visited, 
can  be  deduced  from  Table  1. 

2.4.1  Origin  of  Projects 

•  All  depend  on  ideas  originated  by  working-level  engineers  for 
their  source  of  projects.  The  interaction  of  these  personnel 
with  their  industrial  colleagues  represents  an  important 
contribution  to  this  process.  Approximately  a  third  of  the 
commands  surveyed  have  established  guidelines  that  identify 
critical  needs  or  missions  that  should  receive  priority  support 
from  MANTECH. 
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Table  1.  Observations  to  Date 
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*Fa11  Guidance  Conference. 

**N/A  =  Not  applicable. 
***Year-round,  in-house  conference. 


•  A  few  use  cost  driver  analyses  but  the  emphasis  of  their  use 
appears  to  be  for  planning  top  down  studies  of  large  production 
facilities. 

•  The  majority  of  commands  provide  guidelines  to  their 

subcommands,  which  define  new  mission  requirements  or  critical 
problem  areas  that  need  emphasis  in  allocating  resources  or 
should  receive  priority  in  the  planning  and  selection  of 
projects.  Some  commands  conduct  conferences  and  seminars  to 
publicize  these  needs  to  industry  as  well  as  government  staff. 

2.4.2  Review  and  Evaluation 

The  use  of  peer  review  groups  is  the  core  of  the  evaluation,  and  is 
common  to  all  commands  interviewed.  Further,  the  manner  in  which 
those  groups  are  organized  and  operated  is  quite  similar.  Review 
group  members  are  selected  on  the  basis  of  expertise  in  the 
disciplines  required  for  conduct  of  the  proposed  projects.  A  chairman 
is  usually  appointed  from  the  staff  of  the  MANTECH  unit  of  the 
conmand,  although  in  some  cases  an  executive  review  board  is  convened 
to  review  the  findings  of  the  peer  review  group.  All  such  groups  are 
conducted  informally,  at  which  time  a  consensus  is  reached  on  the 
value  of  each  project  and  those  to  be  selected  for  approval.  The 
consensus  is  based  on  subjective  evaluations  referenced  to  evaluation 
parameters  established  for  each  convnand,  discussion,  and  persuasion. 
None  of  the  commands  has  written  guidelines  or  a  description  of  these 
proceedings,  although  AVSCOM  has  published  a  regulation  (70-6)  that 
defines  policy,  responsibilities,  and  the  eligibility  criteria  for 
projects. 

The  set  of  screening  criteria  and  evaluation  parameters  selected  by 
each  review  group  varies  widely  and  runs  the  gamut  from  the  specific 
(e.g.,  the  savings  to  investment  ratio  shall  exceed  a  given  number)  to 
the  intangible  (e.g.,  improve  the  military  industrial  base). 
Nevertheless,  each  set  contains  elements  that  are  common  to  others, 
and  most  appear  to  be  derived  from  published  regulations  and 
instructions.  Table  2  lists  those  criteria  and  parameters  obtained 
from  the  survey.  In  addition,  the  screening  criteria  and  evaluation 
parameters  contained  in  regulations,  instructions,  and  two 
publications  on  evaluation  methods  are  included  for  comparison  (Ref.  4 
and  6). 

It  is  apparent  from  Table  2  that  regulations,  as  a  group,  prescribe 
more  screening  factors  than  the  commands  follow  in  their  evaluations 
(33  versus  25).  The  criterion  of  generic  application  represents  the 
greatest  difference;  of  the  commands  surveyed,  only  two  listed  this  as 
a  requirement.  However,  it  is  used  as  an  evaluation  parameter  by 
three  of  the  commands  surveyed.  In  one  sense,  emphasis  on  projects 
for  single  products  is  to  be  expected  from  the  way  in  which  most 
projects  are  originated.  The  concerns  of  engineers  are  likely  to  be 
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Table  2.  Criteria  and  Parameters 
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Table  2.  Criteria  and  Parameters  (continued) 
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Table  2.  Criteria  and  Parameters  (continued) 
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Less  Costly 
Insure  Producibi I ity 
Reduce  Lead  Time 
Develop  Multiple  Source 
Improve  Industry  Base 


Table  2.  Criteria  and  Parameters  (continued) 
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Table  2.  Criteria  and  Parameters  (continued) 
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dominated  by  problems  they  identify  with  the  manufacture  of  a 
particular  system  they  are  involved  with  or  are  responsible  for.  In 
most  cases,  they  do  not  have  the  opportunity  or  time  to  stand  back  and 
assess  the  potential  of  their  concept  to  other  applications.  Overall, 
the  most  frequently  cited  screening  criteria  are  that  the  feasibility 
of  the  project  is  proved  and  that  it  does  not  duplicate  previous 
work.  The  most  frequently  cited  evaluation  parameters  are  that  the 
project  is  timely  (i.e.,  for  application)  and  that  it  conserves 
critical  materials.  Direct  questions  regarding  the  need  to  support 
more  than  a  single  weapons  system  elicited  ambiguous  responses.  There 
seems  to  be  hope  that,  although  the  proposed  project  is  directed  to  a 
single  end  item,  as  the  project  develops  it  will  prove  beneficial  to 
the  production  of  other  end  items. 

None  of  these  review  groups  uses  a  numerical  method  to  rank  the 
projects  under  review.  Three  studies  (Ref.  4,  6,  10)  were  conducted 
to  develop  such  methods;  however,  none  has  ever  been  implemented.  The 
fact  that  those  projects  to  be  compared  represent  such  a  wide 
diversity  of  kinds  of  work  and  technical  disciplines  undoubtedly 
contributes  to  the  difficulty  in  finding  a  suitable  alternative  to  the 
current  method.  A  few  participants  are  uncomfortable  with  the 
subjective  basis  of  the  review  group  decisions,  but  most  are 
experienced  in  the  project  areas  under  review  and  are  qualified  in 
their  fields.  It  is  believed  by  most  of  those  interviewed  that  the 
review  groups  can  exercise  informed  judgment  that  provides  a  high 
confidence  in  the  results  and  that  the  encumbrance  of  a  higher  degree 
of  sophistication  or  quantification  is  not  warranted. 

All  Army  and  Navy  commands  require  an  economic  analysis  to  be 
performed  in  accordance  with  AR  11-28  (Ref.  5)  and  NAVMATINST  4800. 36D 
(Ref.  8),  respectively,  but  no  comparable  directives  exist  for  the  Air 
Force.  NMCIRD  does  emphasize  cost  driver  analyses  as  a  tool  for 
planning  projects,  as  does  IBEA.  However,  the  results  of  the  economic 
analyses  do  not  appear  to  be  of  great  importance  during  review  and 
evaluation.  The  review  groups  do  not  normally  question  the  validity 
of  cost  savings  estimated  by  the  proposer  and  possibly  assume  that  a 
successful  project  will  be  cost  effective.  Many  of  those  interviewed 
expressed  skepticism  regarding  the  projection  of  benefits,  or 
financial  savings,  that  would  accrue  from  implementation  of  the 
project  because  they  believe  it  is  extremely  difficult  to  assess  the 
extent  of  technology  transfer  to  be  obtained.  A  few  mentioned  that  a 
project  fulfilling  a  critical  need  would  be  approved  with  a  savings  to 
investment  ratio  of  less  than  one. 

2.4.3  Project  Award 

After  approval  and  release  of  funds,  projects  undergo  the  normal 
procurement  cycle  of  other  programs  that  requires  6  to  9  months. 
There  is  no  consistency  among  commands  regarding  the  ratio  of 
competitive  to  sole-source  awards.  Some  leave  the  decision  up  to 
procurement;  others  appear  divided  between  favoring  competitive  or 
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sole-source  bids.  After  award,  a  project  engineer  is  assigned  to 
monitor  each  project;  this  person  is  usually  a  staff  member  of  the 
command's  MANTECH  group,  although  not  always.  In  addition,  a  few 
commands  hold  quarterly  reviews  of  progress  over  and  above  the  status 
reports  required  by  regulations. 

The  Army  has  the  most  formalized  procedure  for  tracking  the  success  of 
projects  by  means  of  the  annual  Effectiveness  Reports  required  by  IBEA 
for  all  projects  that  have  been  completed.  Also,  MICOM  has  supported 
innovative  methods  to  promote  technology  transfer  from  its  projects. 
The  Air  Force  is  just  beginning  efforts  to  track  the  successes  of  its 
projects.  Efforts  by  the  Navy  in  this  area  are  not  known. 

2.5  Conclusions  From  Findings 

The  procedures  used  by  the  three  services  in  selecting  MANTECH 
projects  are  quite  similar.  All  depend  on  inputs  from  contractors  in 
deriving  new  projects,  and  the  use  of  peer  review  groups  in  evaluation 
is  universal.  The  screening  criteria  and  evaluation  parameters  used 

by  the  various  commands  have  many  common  elements,  although  each  is  a 

unique  set.  In  general,  no  more  than  eight  factors  are  considered 
during  evalution.  None  of  these  commands  has  ever  attempted  to  use 
algorithms  or  quantitative  measures  of  utility  for  ranking  or 

selecting  projects.  The  Army  has  the  most  thorough  procedures  and 
detailed  guidelines  of  the  services  for  processing  and  managing 
MANTECH  projects,  possibly  because  it  has  the  largest  budget  and  most 
experience  in  MANTECH. 

The  applicability  of  these  procedures  to  TACOM  for  selecting  projects 
is  still  under  analysis;  however,  the  following  tentative  conclusions 
are  offered: 

•  The  only  unique  aspect  of  the  TACOM  process,  as  with  any 

command,  is  its  product  line,  i.e.,  combat  and  tactical 
vehicles.  Therefore,  the  use  of  guidelines  that  define  the 

major  thrust  areas  and  critical  needs  for  the  manufacture  of 
these  products  is  recommended. 

•  Obviously,  a  peer  review  group  must  be  used  at  some  level  and 

in  some  fashion  to  rank  and  select  proposals  for  recommendation 
to  IBEA.  The  diversity  of  technologies  implicit  in  the 
projects  submitted  requires  a  matrix  support  to  attain  the 
necessary  broad  range  of  expertise.  The  operation  of  these 
groups  in  the  commands  surveyed  has  been  characterized  as 
"informal"  and  reaching  decisions  by  means  of  "informed 

judgment."  This  process  needs  a  better  definition  to  be  of 

use,  and  alternative  methods  in  the  selection  and  management  of 
the  review  group  are  being  explored.  It  should  be  unique  to 

the  extent  that  it  meets  the  needs  and  resources  of  the  TACOM 

organization. 
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•  The  screening  criteria  and  evaluation  parameters  used  by  those 
commands  surveyed,  and  as  extracted  from  regulations  and  other 
publications,  are  so  numerous  that  they  cover  nearly  every 
utility  aspect  one  could  conceive  of  for  any  project. 
Therefore,  it  is  doubtful  that  factors  identified  for  the 
evaluations  of  TACOM  would  not  be  those  listed  in  Table  2,  or 
at  least  subsumed  under  one  of  the  entries. 

The  identification  of  criteria  and  parameters  is  nearly  complete. 
Screening  criteria  must  be  in  accord  with  Army  regulations  and 
guidelines.  The  evaluation  parameters  should  have  the  following 
characteristics: 

•  Be  applicable  to  a  broad  range  of  projects  covering  processes, 
techniques,  and  equipment  for  both  MMT  and  MAC I; 

•  Be  specific  and  capable  of  being  defined  without  ambiguity  or 
the  requirement  for  lengthy  contingency  clauses; 

•  Be  mutually  exclusive,  which  is  related  to  the  previous 
characteristic;  and 

•  Where  possible,  be  related  directly  to  cost  benefits  or  be 
quantifiable. 

The  selected  list  should  be  made  as  small  as  possible  in  order  to  make 
the  methodology  more  manageable.  This  aspect  is  discussed  further  in 
Section  4.0. 

The  feasibility  of  developing  a  practical  algorithm  for  reaching 
decisions  "by  the  numbers"  is  still  under  consideration  but  the 
experience  of  others  who  have  attempted  this  does  not  make  this 
approach  appear  promising.  Nevertheless,  it  might  be  useful  in 
arriving  at  a  first  cut  to  segregate  candidates  in  groups  prior  to  the 
final  ranking. 

3.0  RESULTS  OF  P-16  EXHIBIT  ANALYSIS 

To  become  familiar  with  the  kinds  of  projects  to  be  evaluated  and 
ranked,  MMT  and  MACI  P-16  exhibits  were  reviewed.  Particular 
attention  was  given  to  FY  1985,  where  each  project  was  tabulated 
according  to  approximately  12  major  headings  (e.g.,  cost,  ROI,  major 
thrust,  MTA6  category,  TACOM  and  IBEA/AMMRC  priority).  The  results  of 
this  analysis  were  inconclusive  in  terms  of  identifying  any  particular 
evaluation  parameters  other  than  the  indication  that  a  high  degree  of 
emphasis  was  placed  on  need  or  problem  solving,  which  was  to  be 
expected.  Also,  there  was  no  positive  correlation  between  the 
IBEA/AMMRC  priority  and  the  ultimate  priority  assigned.  For  example, 
code  31  projects  (tracked  combat  vehicles)  submitted  for  FY  1985 
consisted  of  37  projects;  of  these,  32  were  MMT  projects  and  5  were 
MACI  projects.  Slightly  more  than  50  percent  of  the  MMT  projects  were 
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for  improving  a  current  production  process  or  for  increasing 
productivity;  approximately  25  percent  were  for  developing  a  new 
production  process,  and  another  25  percent  were  for  developing  new 
inspection  or  test  procedures.  Attempts  to  relate  the  IBEA,  AMMRC,  or 
TACOM  rankings  for  these  projects  resulted  in  scatter  diagrams  with  no 
evident  correlation.  The  approval  percentage  in  each  of  these  groups 
obtained  from  IBEA  or  AMMRC  varied  from  50  percent  to  0;  it  is  not 
known  how  these  projects  will  fare  in  the  apportionment  review.  Oust 
under  half  of  the  proposed  projects  are  ongoing  efforts. 

While  efforts  to  find  patterns  or  correlations  related  to  cost 
benefits,  priority  rankings,  evaluation  scores,  or  the  IBEA/AMMRC 
recommendations  were  not  successful,  a  review  of  the  evaluation 
comments  of  IBEA  and  AMMRC  revealed  a  set  of  shortcomings  most 
frequently  cited,  the  distribuion  of  which  is  shown  in  Figure  2.  The 
abscissa  of  Figure  2  indicates  the  issue,  or  area  of  concern,  to  which 
each  set  of  comments  is  addressed.  Each  number  refers  to  a  paragraph 
of  the  P-16  form;  other  captions  refer  to  a  screening  factor  or 
guidelines  on  the  preparation  of  the  P-16  form.  Figure  3  is  a  sample 
P-16  form.  Part  I. 

The  fact  that  some  submittals  did  not  meet  the  definition  of  MMT 
effort  as  given  in  the  regulations  was  the  most  frequent  (38)  comment 
by  far.  The  most  common  basis  for  this  judgment  was  that  the  proposal 
was  a  facilities  project.  Other  reasons  were  that  it  was  an 
industrial  productivity  improvement  or  product  improvement  program 
effort,  or  that  the  proposed  technique  had  been  applied  successfully 
or  was  state  of  the  art. 

The  second  highest  categories  concerned  item  14,  "Key  Milestone 
Dates,"  and  the  environmental  statement  of  Inclosure  1  (20  each). 
Apparently,  the  title  of  item  14  was  assumed  by  originators  to  refer 
to  milestones  within  the  MMT  project  rather  than  the  project 
implementation  milestones;  perhaps  the  title  should  be  changed.  The 
reasons  for  citing  Inclosure  1  were  that  it  was  omitted,  was  not 
signed  off  by  an  environmental  officer,  or  did  not  include  a  Record  of 
Environmental  Consideration.  Other  significant  issues  included; 

•  Item  10a:  Quantifiable  Benefits  (17)— In  13  cases,  none  were 

given;  in  4  cases,  there  was  no  basis  given  for  the 
estimate. 

•  Item  8:  Solution  (16)— The  most  common  reason  for  this 

comment  was  a  lack  of  detail,  followed  by  the 
observation  that  the  solution  did  not  address  the 
problem  or  a  manufacturing  activity. 
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Figure  2.  Distribution  of  Comments 


Date: 


EXHIBIT  P-16  (PART  I) 

Production  Engineering  Measures 
(PEM)  Project 
RCS  DRCMT-835 

I.  Project  Number:  2.  Fiscal  Code:  3.  Cost: 

4.  Project  Title: 

5.  Name  and  location  of  facility/contractor: 

6.  General  Objective: 

7.  Problem 

8.  Solution: 

9.  Justification: 

10.  Benefits: 

a.  Quantifiable  benefits  (S/I): _ Basis: 

b.  Non-quantif iable  benefits: 

II.  Deliverables 

12.  Funding  Profile  and  Scheduled  Technical  Completion  Dates: 
Fiscal  Year  $Costs  Month-Year 


Total 


Figure  3.  P-16  Form 


C-29 


13.  End  Items  Supported: 

a.  Primary 

b.  Secondary 

14.  Key  Milestone  Dates: 

a.  PEP  Completion  for  primary  end  item 

b.  MMT  Completion 

c.  Primary  End  Item  TC 

d.  Start  of  Full  Scale  Production 

e.  Preliminary  Design  Criteria  for  Facility 

15.  Related  MMT  and  Feasibility  Demonstration  Efforts: 

a.  Project  Nos.  _  _  _ 

b.  Initiation  Date  _  _  _ 

c.  Completion  Date  _  _  _ 

16.  Plan  for  Implementation  of  Efforts'  Results: 

a.  When 

b.  Where 

c.  How 

d.  Who 

e.  Cost 

17.  Energy  Resource  Impact  Statement: 


18.  Project  Engineer: 

a.  Name  - 

b.  Organization  - 

c.  Phone  Numbers  AUTOVON _  Commercial 

Inclosure  1  -  Environmental  Documentation 


Figure  3.  P-16  Form  (continued) 


•  Item  15:  Related  Efforts  (15)— Here,  the  most  common  comments 

were  that  the  referenced  project  was  not  valid  (i.e., 
terminated),  further  detail  of  the  relationship  of  the 
proposed  effort  to  the  related  effort  was  desired,  or 
additional  related  efforts  were  suggested.  In  one 
case,  it  was  stated  that  the  results  of  previous  work 
on  the  project  (i.e.,  for  an  ongoing  effort)  should  be 
known.  While  this  may  be  valid,  there  does  not  appear 
to  be  a  place  on  the  P-16  form  for  summarizing  the 
status  of  previous  work. 

R&D  (14)--This  refers  to  the  screening  factor  that 
states  MMT  projects  cannot  support  research  and 
development  work.  In  5  of  these  cases,  the  conclusion 
that  the  proposal  involved  research  and  development 
was  in  doubt. 

•  Item  6;  General  Objective  (13) — The  objective  stated  was  not 

included  in  the  approved  list  contained  in  the  P-16 
guidel ines. 

•  Item  7:  Problem  (ll)--Two  main  objections  were  noted:  either 

more  detail  was  needed  or  it  was  not  a  manufacturing 
problem. 

•  Item  12:  Funding  Profile  (9)--These  comments  concerned 

inconsistencies  within  the  P-16  or  other  data  or  that 
the  costs  were  too  high. 

•  Item  16:  Implementation  Plan  (9)— The  most  frequent  comment  was 

lack  of  cost  data  on  implementation;  in  two  other 
cases,  MMT  costs  rather  than  implementation  costs  were 
included. 

•  Item  11:  Deliverables  (7)— A  general  lack  of,  or  clarity  in, 

the  identification  of  deliverables  was  cited  here. 

•  Item  9:  Justification  (7)— More  detail  or  making  a  better  case 

was  desired. 

P-16  (6)--In  this  category,  the  comment  was  that 

system  projects  with  more  than  one  task  require  an 
overall  P-16  plus  separate  P-16s  for  each  task. 

DUPE  (4)--Four  cases  of  duplicating  previous  work  were 
found. 

In  addition,  frequent  reference  was  made  to  the  limit  of  two  pages  for 
each  Part  I  P-16  form  submittal.  The  conclusions  of  this  review  are 
that  the  submitters  should  be  reminded  of  what  does  and  does  not 
constitute  MMT  effort,  and  that  the  P-16  guidelines  should  be  at  hand 
when  preparing  submittals. 


C-31 


4.0  EVALUATION  PARAMETERS 


4.1  Introduction 

All  MANTECH  projects  are  designed  to  satisfy  an  established  Army  need 
to  improve  a  particular  manufacturing  process,  technique,  or  equipment 
in  a  manner  that  has  broad  applicability  within  the  Army  as  well  as  to 
other  military  branches  and  private  industry.  This  need  is  derived 
from  experience  with,  or  analysis  of,  ongoing  manufacture  and 
maintenance  of  existing  weapon  systems,  as  well  as  the  requirements 
imposed  by  new  weapon  systems  entering  the  acquisition  cycle.  Typical 
candidate  areas  for  the  application  of  MANTECH  resources  are: 

•  Labor-intensive  operations; 

•  Operations  that  generate  high  waste,  particularly  of  scarce  and 
high-cost  material; 

•  Complex  operations  that  can  be  consolidated,  eliminated,  or 
streamlined; 

•  Low  production  yield,  where  improvements  will  reduce  rejection 
rates; 

•  Dangerous  manufacturing  operations,  e.g.,  toxic  materials 
handling,  explosives,  pollutants,  and  equipment;  and 

•  General,  high-cost  areas,  other  than  those  cited  above,  where 
it  is  known  that  a  particular  manufacturing  phase  represents 
such  a  large  percentage  of  costs  that  any  improvement 
(management-material-process)  will  have  a  payoff  in  dollars 
saved. 

Once  a  candidate  area  is  identified  as  having  the  potential  for  the 
application  of  MANTECH  resources,  proposed  manufacturing  changes 
should  be  screened  to  insure  that  they  meet  the  criteria  for  a  MANTECH 
project.  In  brief,  these  criteria  include  the  following: 

•  Does  it  address  new  or  improved  manufacturing  technology  that 
has  not  been  applied  previously? 

§  Has  feasibility  been  demonstrated  (laboratory,  bench  scale,  or 
extrapolation  of  experimental  data)? 

•  Is  it  production  oriented,  with  the  potential  for  broad/ 
generic  application? 

•  Should  private  industry  pursue  it  on  its  own,  or  is  it  within 
the  normal  risk  that  industry  might  take? 
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•  Is  it  likely  that  it  might  be  available  within  the  next  5  to  7 
years  without  TACOM  sponsorship? 

•  Is  it  likely  to  reduce  costs? 

•  Does  it  duplicate  any  past  or  ongoing  MANTECH  projects? 

•  Is  it  definitely  related  to  manufacture  of  the  end  item  as 
opposed  to  the  end  item  itself? 

4.2  Identification  of  Parameters 

It  is  logical  that  a  number  of  projects  will  be  identified  that 
satisfy  the  initial  screening  criteria;  however,  their  total  cost  may 
exceed  the  financial  resources  available.  Furthermore,  it  is  always 
desirable  to  prioritize  projects  to  insure  that  the  appropriate  amount 
of  effort  is  devoted  to  the  projects  that  will  provide  the  greatest 
overall  benefit.  For  these  reasons,  each  competing  project  should  be 
evaluated  further.  On  this  basis,  need  and  cost  savings  obviously 
stand  out  as  two  important  parameters  for  evaluating  competing  MANTECH 
projects.  These  two  parameters,  as  well  as  others,  can  be  grouped 
logically  in  areas  that  relate  to  each  other: 

•  Criticality/Need 

--  Increased  Production 
—  Advancement  in  State  of  the  Art 
--  Conservation  of  Critical  Material 

•  Cost/Benefit 

•  Quality 

—  Reliability 
—  Maintainability 
—  Availability 
—  Durability 

•  Regulatory/Other 
—  Safety 

—  Reduced  Pollution 
“  Energy  Conservation 

•  Risk 

--  Likelihood  of  Success 
—  Likelihood  of  Implementation 

4.2.1  Critical ity/Need 

Criticality/need  can  be  rated  independently.  It  is  the  most  important 
parameter  in  evaluating  TACOM  MANTECH  projects.  It  focuses  on  solving 
existing  problems,  or  providing  solutions  for  potential  problems.  It 
is  almost  an  umbrella  that  covers  all  other  evaluation  parameters, 
whether  they  be  objective  or  subjective. 
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Existing  manufacturing  bottlenecks,  new  system  designs,  new  system 
performance  requirements,  mission  support  requirements  (readiness, 
mobility)  material  shortages,  and  maintenance  problems,  contribute  to 
the  creation  of  a  need  for  MANTECH  projects.  Even  the  criterion  of 
economics  can  be  expressed  as  a  need  (e.g.,  the  need  to  reduce  costs), 
although  it  is  used  more  generally  as  a  measure  of  merit  in  evaluating 
competing  projects. 

All  projects  should  respond  to  a  need,  and  the  criticality  of  the  need 
is  a  function  of  how  well  the  current  manufacturing  process  is  suited 
to  producing  a  weapon  system  or  commodity  end  item  that  meets  its 
technical  performance  or  design  requirements  in  a  timely  and 
economical  manner.  Is  the  proposed  change  really  an  improvement?  Is 
it  really  needed?  How  urgent  is  it?  Could  it  be  used  now  if  it  were 
available? 

4. 2. 1.1  Increased  Production 

Increased  production  is  not  to  be  rated  independently;  it  is  to  be 
incorporated  in  cost  savings  as  a  part  of  the  economic  analysis  or 
criticality/need.  One  should  differentiate  between  increases  in  the 
number  of  units  produced  at  a  given  production  rate,  and  the  number  of 
units  produced  due  to  increasing  the  rate  of  production.  In  the 
former  case,  the  increase  may  be  achieved  by  adding  a  second  line  or 
bringing  another  contractor  on  board,  either  of  which  may  not  qualify 
it  as  a  MANTECH  project.  The  second  case  of  increasing  the  rate  of 
production  has  to  be  the  result  of  an  improvement  in  process, 
technique,  or  equipment. 

Although  increased  production  is  normally  a  laudible  goal,  the 
question  has  to  be  asked,  "Why?"  In  general,  it  can  be  for  one  of  two 
reasons.  Either  the  current  rate  is  inadequate  to  meet  requirements 
reflected  in  the  current  Mission  Statement,  Army  Material  Plan,  or 
other  appropriate  Army  document,  or  costs  can  be  reduced  by  producing 
more  units  (if  required)  in  a  given  time,  or  a  given  number  of  units 
in  less  time. 

The  first  reason,  as  discussed  in  the  first  paragraph  above,  would 
probably  not  qualify  it  as  a  MANTECH  project  even  if  another  source  of 
supply  were  required  to  provide  insurance  in  the  case  of  crisis. 

The  second  reason  may  be  reflected  in  the  "criticality/need" 
parameter,  and  the  benefits  will  certainly  be  included  in  the 
"cost/benefit"  parameter. 

4. 2. 1.2  Advancement  in  State  of  the  Art 


Advanced  state  of  the  art  can  be  rated  independently  if  it  is  the  only 
purpose  and  not  covered  by  criticality/need.  Advances  in  technology 
purely  for  the  sake  of  technology  are  difficult  to  justify.  Without 
an  expressed  need,  there  is  a  small  likelihood  of  implementation  and 
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these  two  parameters,  need  and  likelihood  of  implementation,  are  very 
important  for  MANTECH  projects.  Furthermore,  technological  advances 
with  no  direct  or  immediate  application  are  more  likely  to  be  in  their 
feasibility  stages,  which  may  exclude  them  from  the  TACOM  MANTECH 
Program.  Although  this  parameter  can  be  quantified  objectively,  it 
will  probably  be  seldom  used  and  one  is  cautioned  not  to  give  it 
double  credit,  i.e.,  if  the  "need"  is  to  "advance...,"  rate  the 
project  under  need  and  not  under  both  "need"  and  "advancement." 

4. 2. 1.3  Conservation  of  Critical  Materials 


Conservation  of  critical  materials  is  not  to  be  rated  independently; 
it  is  covered  by  criticality/need.  The  need  to  conserve  critical 
materials  or  resources  can,  by  itself,  be  a  justification  for  a 
MANTECH  project.  However,  this  would  be  covered  under  the  parameter 
of  criticality/need.  A  MANTECH  project  is  a  solution  to  the  need  and 
it  may  embody  the  substitution  of  new  or  different  materials,  which 
may  dictate  a  change  in  the  manufacturing  process,  or  just  a  change  in 
the  manufacturing  process  that  eliminates  waste. 

4.2.2  Cost/Benefit 

Cost/benefit  can  be  rated  independently.  Cost/benefit  can  be  viewed 
from  two  aspects:  (1)  reducing  costs  and  (2)  using  reduced  costs  in 
evaluating  competing  projects.  In  the  first  instance,  either  on  the 
basis  of  intuitive  observation  or  something  more  sophisticated  like 
"cost  driver  analysis,"  a  project  is  promoted  purely  to  reduce  costs. 
Reducing  cost  then  becomes  a  "need,"  and  one  should  establish  a  goal 
of  how  much  costs  reduction  is  desired. 

More  than  likely,  the  project  will  be  promoted  based  on  one  or  more  of 
the  other  quantifiable  evaluation  parameters.  In  this,  the  second 
aspect,  cost/benefit  becomes  a  measure  of  how  well  one  proposed 
project  compares  with  another.  Because  it  is  possible  to  derive  a 
quantitative  measure  of  merit,  a  comparison  appears  straightforward. 
However,  two  things  should  be  taken  into  consideration:  (1) 
limitations  on  the  answer  and  (2)  various  ways  in  which  the  answer  can 
be  derived. 

With  respect  to  limitations  on  the  answer,  it  is  important  to  use 
"life  cycle"  costs  in  deriving  the  economic  analysis,  as  there  is 
often  the  temptation  to  look  only  at  an  immediate  return  on 
investment,  which  can  prove  to  be  small.  It  also  would  be  beneficial 
to  look  at  "net"  savings  based  on  either  the  complete  end  product  or 
total  weapon  system.  Complete  reliance  on  the  savings  to  investment 
ratio  can  lead  to  an  emphasis  of  incremental,  or  percentage,  savings 
that  ignores  larger  absolute  savings,  particularly  when  the  absolute 
savings  are  small  in  proportion  to  investment.  Its  easy  to  forecast  a 
large  incremental  savings  for  a  single  project  that  may  only  be  a 
small  segment  of  the  total  production  cycle  and,  therefore, 
insignificant  in  the  total  picture,  while  at  the  same  time  passing  up 
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a  smaller  incremental  savings  from  another  project  that  would  really 
be  more  significant  in  the  total  picture. 

With  respect  to  number  of  ways  the  answer  can  be  determined,  there  are 
a  number  of  economic  measures  of  merit  available  to  the  analyst. 
These  are  discussed  in  detail  in  Section  5.0. 

4.2.3  Quality 

Quality  is  not  to  be  rated  or  considered  independently  because  it  is 
covered  by  other  parameters.  Quality  is  really  the  umbrella  of  all 
the  other  "ilities."  It  is  difficult  to  envision  something  that  is 
highly  reliable,  very  durable,  and  easy  to  maintain,  and  yet  is  of 
poor  quality.  Also,  if  it  has  all  these  good  attributes,  it  most 
certainly  should  enhance  availability.  Yet,  "improved  quality"  is 
often  cited  as  the  justification  for  a  MANTECH  project.  Furthermore, 
the  reference  is  usually  to  the  quality  of  the  end  item  being 

produced--not  the  quality  of  the  manufacturing  change.  This  approach 
is  not  entirely  valid,  and  the  following  distinctions  should  be  made. 

Quality  must  be  "designed  into"  the  hardware,  whether  it  be  the  end 
item  being  produced  or  the  process,  technique,  or  equipment  that 
produces  it.  There  is  no  incentive  or  real  payoff  in  "improving" 
quality  beyond  that  required  by  the  design.  What  is  required  is  to 
"insure"  that  the  designed  quality  is  being  met  and  maintained. 

The  primary  focus  in  evaluating  MANTECH  projects  should  be  on  the 
quality  of  the  manufacturing  process,  technique,  or  equipment.  Test 
and  inspection  of  this  improvement  change*  can  only  demonstrate  the 

degree  to  which  quality  is  being  maintained.  It  would  seem  quite 
natural  that  a  poor  quality  manufacturing  method  is  likely  to  produce 
a  poor  quality  end  item.  Therefore,  in  the  event  one  starts  to  see  a 
degradation  in  the  quality  of  the  end  item,  it  may  be  wise  for  the 
hardware  design  engineer  to  collaborate  with  the  production  design 
engineer  and  ascertain  the  true  cause.  This  definition,  in  turn,  may 
dictate  a  new  MANTECH  need;  the  solution  for  which  will  again  be 

evaluated  according  to  other  ilities. 

A  particular  MANTECH  project  under  evaluation  may  be  the  direct  result 
of  a  "need"  to  insure  quality  of  the  end  item.  More  sophisticated 
machinery,  tighter  controls,  and  increased  automation  are  examples  of 
changes  that  can  increase  precision  and  guarantee  reproducibility  or 
repeatability,  thereby  increasing  yield.  Quite  often,  however,  these 


♦There  also  are  end  item  peculiar  "test  and  inspection"  MANTECH 
projects  that  should  not  be  confused  with  the  test  and  inspection  of 
the  new  manufacturing  change.  MANTECH  test  and  inspection  projects 
represent  improvements  in  such  areas  as  better  screening,  easier 
testing,  and  faster  screening  or  testing. 
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projects  are  promoted  under  the  military  Quality  Assurance  Progam, 
which  is  a  separate  DOD  activity. 

4.2.3. 1  Reliability 

Reliability  is  not  to  be  rated  independently;  it  is  incorporated  in 
cost  savings,  which  is  part  of  the  economic  analysis.  Reliability  is 
often  reflected  in  terms  that  relate  to  the  performance  of  the  end 
item  being  produced  and  not  the  manner  by  which  it  is  produced. 
Producing  in  a  manner  that  causes  poor  quality  or  high  rejection  rates 
does  not  affect  the  reliability  of  the  end  item  which,  if  built  to 
design,  would  have  the  same  inherent  reliability;  it  just  makes  the 
required  quality  more  difficult,  time  consuming,  and  costly  to 
achieve.  For  MANTECH  projects,  however,  the  measure  of  reliability 
should  be  restricted  to  the  manufacturing  process,  technique,  or 
equipment;  mechanisms  that  are  not  large  repetitive  volume  buys  and 
that  are  not  themselves  subjected  to  lot  sampling  or  reliability 
testing,  although  some  of  the  piece  parts  that  go  into  these 
mechanisms  may  be  so  tested. 

By  definition,  if  the  manufacturing  technology  project  is  to  be  an 
improvement,  it  cannot  degrade  reliability,  and  the  degree  of 
improvement  is  very  difficult  to  measure.  Therefore,  the  best  measure 
of  a  "more  reliable"  manufacturing  process,  technique,  or  equipment 
would  be  an  increase  in  production  rate  resulting  from  either  less 
down  time  (as  mechanism  is  repaired)  or  lower  rejection  rates  (higher 
yield).  A  new  mechanism  that  just  produces  at  a  faster  rate  would  not 
represent  an  increase  in  reliability. 

4. 2. 3. 2  Maintainability 

Maintainability  is  not  to  be  rated  independently;  it  is  covered  by 
economic  analysis.  By  definition,  any  improvement  in  a  manufacturing 
process,  technique,  or  piece  of  equipment  is  not  an  improvement  if  it 
degrades  maintainability.  Again,  one  has  to  be  careful  to  distinguish 
between  maintenance  of  the  production-oriented  improvement,  as  opposed 
to  maintenance  of  the  end  item  being  produced.  For  MANTECH  projects, 
the  emphasis  should  be  on  maintenance  of  the  production  change.*  The 
design  of  the  end  item  being  produced,  as  opposed  to  the  manner  in 
which  it  is  manufactured,  normally  affects  the  maintenance  of  the  end 
item. 


*There  are,  however,  some  MANTECH  projects  that  can  enhance  mainte¬ 
nance  of  the  end  item.  For  example,  certain  improved  "test  and 
evaluation"  methods  or  equipment  may  be  better  able  to  predict 
failures,  thereby  extending  the  useful  life  of  the  end  item,  while 
still  allowing  for  corrective  action  before  the  failure  actually 
takes  place. 
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Measures  of  the  maintainability  of  the  production  change  are  reflected 
in  attributes  that  contribute  to  the  ease  of  repair,  the  extent  to 
which  it  must  be  replaced,  and  how  long  will  it  last  (mean  time  to 
repair  or  replace).  These  are  life  cycle  questions  that  apply  to  the 
manufacturing  change  and  not  the  end  product.  Any  possible  negative 
impact  on  maintainability  of  the  end  item  would  be  grounds  for 
canceling  the  project  and,  it  is  hoped,  these  relationships  would  have 
been  defined  in  the  R&D  or  feasibility  stage  prior  to  becoming  a 
MANTECH  project. 

These  measures  should  be  reflected  in  the  economic  analysis  as  they 
are  related  directly  to  manufacturing  operations  and  maintenance  (O&M) 
costs  and  the  costs  of  varying  the  production  time  and  rate. 

4. 2. 3. 3  Availability 

Availability  is  not  to  be  rated  independently;  it  is  covered  by 
economic  analysis.  As  with  maintainability,  an  improved  manufacturing 
process,  technique,  or  equipment  is  not  an  improvement  if  it  is  not 
available.  A  highly  precise  and  repeatable  piece  of  machinery  that  is 
unreliable  increases  downtime.  The  emphasis  for  MANTECH  projects 
should  be  on  the  availability  of  the  improvement,  which  is  largely  a 
function  of  its  reliability  and  maintainability,  or  MTBF  and  MTTR. 

The  case  where  there  just  "aren't  enough"  of  an  improved  piece  of 
manufacturing  equipment  (i.e.,  unavailability)  is  an  insufficient 
reason  to  promote  it  as  a  MANTECH  project  and,  therefore,  should  not 
be  considered.  The  acquisition  of  additional  normal  manufacturing 
capability  to  expand  the  industrial  base  or  to  meet  surge/mobilization 
requirements  should  be  covered  by  normal  production  procurement  funds. 

4. 2. 3. 4  Durability 

Durability  is  not  to  be  rated  independently;  it  is  covered  by  economic 
analysis.  Again,  as  with  other  ilities,  one  should  assess  the 
durability  of  the  improved  manufacturing  process,  technique,  or 
equipment  not  the  durability  of  the  end  item  being  produced.*  Will 
the  new  mold,  die,  etc,,  hold  up  as  well  as  the  item  it  is  replacing, 
or  will  the  automated  handling  equipment  wear  out  before  its  manual 
counterpart? 


♦There  are,  however,  some  MANTECH  projects  that  can  enhance  dura¬ 
bility  of  the  end  item.  For  example,  a  change  to  a  less  corrosive 
material,  or  a  new  rubber  compound  that  makes  the  end  item  more 
durable,  may  require  a  new  manufacturing  method  to  produce  it. 
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4.2.4  Regulatory/Other 


4. 2.4.1  Safety 

Safety  can  be  rated  independently.  Compliance  with  OSHA  regulations 
is  a  normal  TACOM  requirement,  and  manufacturing  changes  to  accomplish 
this  are  normally  funded  with  operations  and  maintenance.  Army  (OMA) 
or  program  funds.  Given  the  other  higher  priority  needs  of  TACOM,  it 
is  unlikely  that  a  MANTECH  project  would  be  approved  solely  on  the 
basis  of  this  parameter.  The  benefit  of  improved  safety,  which  may  be 
quantifiable,  would,  by  itself,  probably  not  be  significant  enough  to 
distinguish  the  project  from  others.  Therefore,  it  is  recommended 
that  this  be  a  secondary  benefit,  which  if  it  occurred,  might  be 
helpful  in  rating  two  competing  projects  that  were  otherwise  equal  in 
all  respects. 

One  also  is  cautioned  not  to  give  double  credit  for  this  parameter, 
i.e.,  if  the  principal  purpose  of  the  project  is  the  "need  to  improve 
safety,"  then  rate  it  according  to  need  and  not  need  plus  additional 
credit  for  safety.  If  the  sole  purpose  of  the  project  is  to  improve 
safety,  it  may  be  possible  to  identify  some  financial  savings  due  to 
increased  productivity  (fewer  personnel  lost  work  days)  or  lower 
overhead  or  burden  rates  (lower  insurance  or  other  compensation). 
This  type  of  information  should  be  reflected  in  the  economic  analysis. 

4. 2. 4. 2  Reduced  Pollution 

Reduced  pollution  can  be  rated  independently.  Compliance  with  ERA 
regulations  is  a  normal  TACOM  requirement,  and  manufacturing  changes 
to  accomplish  this  are  normally  funded  with  OMA  or  program  funds. 
Given  other  higher  priority  needs  of  TACOM,  it  is  unlikely  that  a 
MANTECH  project  would  be  approved  solely  on  the  basis  of  this 
parameter.  The  benefits  of  reduced  pollution  are  hard  to  quantify  as 
they  are  a  function  of  the  type  of  pollutant,  the  geographic  area  of 
the  source,  the  amount  of  pollutant  being  discharged,  and  the 
environmental  pathways  for  its  impact.  Therefore,  competing  projects 
cannot  be  separated  based  only  on  this  parameter.  Furthermore,  quite 
often  the  incorporation  of  pollution-reduction  equipment  will  cost 
more,  so  a  savings  to  investment  ratio  calculation  would  not  be  a 
reason  to  promote  the  project. 

Therefore,  it  is  recommended  that  this  be  a  secondary  benefit,  which, 
if  it  occurred,  might  be  helpful  in  rating  two  competing  projects  that 
were  otherwise  equal  in  all  respects.  One  also  is  cautioned  not  to 
give  double  credit  for  this  parameter,  i.e.,  if  the  principle  purpose 
of  the  project  is  "need  to  reduce  pollution,"  rate  it  according  to 
need  and  not  need  plus  additional  credit  for  reducing  pollution. 
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4. 2. 4. 3  Energy  Conservation 

Energy  conservation  is  not  to  be  rated  independently;  it  is  covered  by 
economic  analysis.  The  need  to  conserve  energy  is  an  important  Army 
program  given  special  consideration  by  the  authority  of  the  "Army 
Energy  Program,"  AR  11-27  (Ref.  11).  Energy  savings  can  be  realized 
through  new  breakthroughs  in  manufacturing  improvement,  that  would 
therefore  make  them  candidates  for  MANTECH  projects.  In  most 
instances,  however,  energy  conservation  projects  are  of  such  a  general 
nature  that  they  are  more  appropriately  funded  with  OMA  funds. 

Given  other  higher  priority  needs  of  TACOM,  the  opportunity  for  a 
project  to  be  supported  solely  on  the  basis  of  energy  conservation  is 
quite  small.  However,  all  MANTECH  projects  present  the  opportunity 
for  conserving  energy  and  the  projected  savings  in  cost  (which  is  the 
best  measure  of  the  benefit,  as  opposed  to  Btu's  or  gallons  of  oil) 
can  be  developed  readily  and  should  be  incorporated  into  the  economic 
analysis. 

4.2.5  Risk 

Risk  may  be  rated  independently,  but  only  to  the  granularity  of  high, 
low,  or  medium.  Risk  is  an  estimate  of  the  degree  of  probability  that 
a  given  MANTECH  project  may  not  achieve  its  stated  purpose(s).  In 
essence,  it  is  merely  a  reflection  of  the  uncertainty  band  around  key 
evaluation  parameters.  For  example,  "Likelihood  of  Success"  assumes 
an  appreciation  of  the  chance  that  the  project's  technical  feasibility 
will  not  prove  worthy  of  incorporation  into  full-scale  production. 
Likewise,  even  if  technical  feasibility  is  proven,  there  is  always 
some  risk  that  the  improvement  may  not  be  implemented  for  reasons  that 
may  be  beyond  the  control  of  the  MANTECH  organization.  This  again 
reflects  an  uncertainty  associated  with  the  evaluation  parameter 
"Likelihood  of  Implementation". 

The  economic  analysis  evaluation  parameter  is  quite  likely  to  contain 
some  risk  because  financial  estimates  are  merely  projections  at  this 
stage.  Furthermore,  this  risk  may  change  with  time--multi-year 
projects  produce  new  data  each  year. 

There  is  a  positive  correlation  between  "technical  risk"  and  "economic 
risk."  The  greater  the  technical  risk,  the  greater  the  likelihood 
that  (1)  it  may  cost  more  than  projected,  even  if  successful,  and  (2) 
that  money  may  be  provided  continually  to  the  project  in  pursuing 
modifications  that  may  make  it  successful,  when  in  reality,  it  should 
perhaps  be  dropped. 

4.2.5. 1  Likelihood  of  Success 


Likelihood  of  success  can  be  rated  independently.  Proposed  projects 
should  present  practicable  alternatives  to  manufacturing  processes, 
techniques,  or  equipment.  The  proposed  project  must  have  demonstrated 
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feasibility  (e.g.,  laboratory  or  bench  scale)  prior  to  undertaking  it 
as  a  MANTECH  project.  However,  there  are  still  problems  that  may  be 
encountered  as  scale-up  to  production  is  not  always  as  straightforward 
as  one  would  like  it  to  be.  Yet,  the  project  must  demonstrate  that 
the  concept  is  suitable  to  direct  transition  to  the  mass-production 
manufacturing  floor.  This  parameter  is  often  expressed  in  terms  of 
technical  risk. 

4. 2. 5. 2  Likelihood  of  Implementation 

Likelihood  of  implementation  can  be  rated  independently.  A  committed 
sponsor  who  is  willing  to  implement  the  proposed  MANTECH  project  is  a 
very  important  asset  to  the  proposal.  Although  the  cost  of  the 
MANTECH  demonstration  is  absorbed  by  separate  funding,  the  benefits 
will  accrue  to  the  Project  Manager  (PM)  responsible  for  the  end  item 
commodity/weapon  system.  There  can  be  problems  with  implementation 
even  if  the  program  office  of  the  end  item  affected  strongly  supports 
a  MANTECH  project  when  it  is  proposed.  After  the  project  is  completed 
and  ready  to  be  implemented,  the  management  of  the  program  office  may 
have  changed.  There  is  then  a  good  chance  that  the  new  PM  may  not  be 
willing  to  accept  the  risk  associated  with  "doing  things  differently 
than  in  the  past."  This  requires  placing  continuing  emphasis  on 
implementation  and  coordination  with  the  program  office.  So,  emphasis 
should  be  placed  on  an  "Implementation  Plan,"  and  a  high  degree  of 
confidence  that  the  improvement  will  be  implemented. 

There  also  is  the  chance  that  the  improvement  would  be  implemented 
without  the  benefit  of  MANTECH.  In  many  instances  the  length  of  the 
budget  cycle  (2  to  3  years),  the  length  of  the  demonstration  project 
(up  to  3  years),  and  the  lead  time  to  get  the  proven  demonstration 
implemented  (1  year  estimated)  are  such  that  events  may  overtake  the 
idea  and  it  may  become  a  reality  without  MANTECH.  This  raises  the 
question,  "What  is  the  likelihood  of  the  project  being  implemented 
without  Army  sponsorship?",  a  question  appropriate  to  initial 
screening  of  MANTECH  proposals. 

5.0  ECONOMIC  ANALYSIS 

The  calculation  of  economic  measures  of  merit  provides  TACOM  with  a 
means  of  comparing  the  benefits  of  independent,  unrelated  MANTECH 
projects.  This  comparison  is  essential  in  prioritizing  MANTECH 
projects  and  in  identifying  the  most  efficient  allocation  of  limited 
financial  resources. 

Because  of  the  somewhat  a  typical  investment  decision  perspective  of 
TACOM  (for  example,  the  orientation  toward  cost  savings  as  opposed  to 
profit  maximization,  differing  time  frames  for  investment  versus 
realization  of  benefits  from  that  investment,  and  substantial 
differences  in-  scale  of  investment  alternatives),  the  range  of 
appropriate  (for  TACOM  use)  measures  of  merit  is  somewhat  limited. 
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Measures  of  merit  are  presented  in  terms  of  ease  of  calculation,  data 
requirements,  interpretation,  strengths  and  weaknesses  of  specific 
analytical  approaches,  and  overall  usefulness  to  TACOM. 

5.1  POD  and  Army  Economic  Analysis  Regulations  and  Directives 

Within  POD,  economic  analysis  and  program  evaluation  for  resource 
management  are  addressed  specifically  in  POD  Instruction  7041.3  (Ref. 
12).  This  instruction  outlines  policy  guidance  and  establishes  a 
DOD-wide  framework  for  consistent  application  of  economic  analysis  on 
proposed  programs,  projects,  and  activities.  The  fundamental 
rationale  behind  the  DOD-wide  emphasis  on  economic  analysis  is  one  of 

having  to  constantly  make  decisions  as  to  how  scarce  resources  should 

best  be  used,  and  to  identify  the  implications  of  achieving  a  given 

objective  in  the  most  efficient  and  effective  manner.  Economic 

analysis  is  used  to  support  decisions  concerning  trade-offs  between 
alternatives,  to  identify  the  most  cost-effective  alternative  when  a 
range  of  alternatives  exists,  and  to  serve  as  a  basis  for 

establishment  or  change  in  priorities. 

Specific  to  TACOM,  AR  11-28  (Ref.  5)  is  consistent  with  POD 

Instruction  7041.3  (Ref.  12).  AR  11-28  establishes  policy, 
procedures,  and  responsibilities  for  the  application  of  economic 
analysis  to  ongoing  as  well  as  proposed,  programs,  projects,  and 
activities.  AR  11-28  defines  economic  analysis  as  the  use  of  a 
systematic  approach  to  the  problem  of  choosing  how  to  employ  scarce 
resources  and  an  investigation  of  the  full  implications  of  costs  and 
benefits  associated  with  alternative  programs,  missions,  or 
alternative  ways  of  accomplishing  a  given  program.  In  the  analysis, 
key  variables  are  to  be  highlighted  that  are  either  identified  as 

highly  sensitive  or  that  have  the  greatest  effect  on  analytical 
results.  The  overall  purpose  of  the  analysis  is  to  systematically 

examine  all  feasible  alternatives,  rank  alternatives  on  the  basis  of 
benefit-to-cost  relationships,  and  assist  the  decisionmaker  in  the 
identification  of  the  most  cost-effective  alternative. 

The  essential  elements  for  any  economic  analysis,  as  defined  in 
AR  11-28  (Ref.  5),  include: 

•  Identification  of  the  mission-related  objectives  of  the 
action(s)  being  considered; 

•  Specification  of  analytical  assumptions  and  constraints; 

t  Listing  of  alternatives; 

•  Listing  of  benefits  (outputs)  for  all  feasible  alternatives; 

•  Estimation  of  costs  for  all  feasible  alternatives;  and 
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•  Presentation  of  the  relationship  of  benefits  to  cost  of 
alternatives  considered  in  the  order  of  their  respective 
economic  performance. 

Where  appropriate,  the  analysis  also  will  contain  an  assessment  of  the 
relative  risk  or  uncertainty  of  success  associated  with  each  of  the 
alternatives  considered.  Benefit  estimates  developed  for  each 
alternative  are  indicators  of  objective  achievement  and  should  be 
expressed  quantitatively  where  possible.  The  analysis  also  should 
include  important  nonquantif iable  benefits  such  as  health,  safety, 
security,  and  energy,  if  these  factors  are  pertinent  to  a  decision 
between  alternatives.  AR  11-28  (Ref.  5)  states  specifically  that 
measures  of  performance  and  productivity  indexes  will  be  used  to  the 
maximum  extent  possible.  In  this  context,  the  measures  of  economic 
merit  (performance),  discussed  in  Section  4.0  of  this  report,  conform 
to  the  requirements  in  AR  11-28. 

Of  all  the  various  elements  and  processes  that  must  be  included  in  the 
economic  analysis,  the  accurate  identification  of  benefits  (outputs) 
is  the  most  difficult.  The  analysis  should  concentrate  on  this  aspect 
because  estimates  of  project  cost  and  timing  are  generally  more 
rigorous. 

5.2  Economic  Measures  of  Merit 


5.2.1  Discounted  Cash  Flow  Techniques 

Discounted  cash  flow  (DCF)  techniques  incorporate  explicit 
consideration  of  the  time  value  of  money,  discounting  future  receipts 
and  disbursements  back  to  the  present  under  an  assumed  discount  rate 
(usually  equivalent  to  the  cost  of  capital).  DCF  techniques  reflect 
the  fact  that  a  dollar  today  is  worth  more  than  a  dollar  1  year  from 
now;  this  is  true  because  the  dollar  today  may  be  invested  so  as  to 
produce  earnings  during  the  intervening  year.  DCF  techniques  enable 
decisionmakers  to  compare  equivalent  estimates  for  cash  flows  of 
alternative  investments  at  the  same  point  in  time. 

The  most  widely  used  DCF  techniques  include  net  present  value  (NPV), 
internal  rate  of  return  (IRR),  profitability  index  (PI),  and 
saVings/investment  ratio  (SIR).  These  DCF  techniques  can  be  further 
broken  down  into  absolute  (NPV,  IRR)  and  relative  (PI,  SIR)  measures 
of  merit.  Absolute  measures  of  merit  indicate  the  actual  benefit 
obtainable  from  a  given  investment,  whereas  relative  measures  of  merit 
indicate  the  ratio  between  the  discounted  receipts  and  disbursements 
or  the  profitability  per  dollar  cost  of  the  investment.  Thus,  the  PI 
and  SIR  are  benefit/cost  ratios. 
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5.2.2  Absolute  Measures  of  Merit 


5.2.2. 1  Net  Present  Value 


The  NPV  for  an  investment  alternative  is  the  present  value  equivalent 
of  its  net  annual  cash  flow  (annual  receipts  less  annual  operating 
costs)  minus  the  present  value  equivalent  of  its  investment  costs.  In 
other  words,  NPV  represents  the  present  equivalent  net  profit  for  an 
investment.  Mathematically,  NPV  can  be  determined  from  the  following 
equation: 
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where  F-^  =  Net  flow  of  dollar  benefits  (or  cost  savings)  in  year 

t; 


K  =  Discount  rate; 


=  Capital  investment  outlay  in  year  t  (t=0  indicates 
project  initiation);  and 

N  =  Lifetime  of  project,  in  years. 


Note  that  the  investments  as  well  as  the  benefits,  or  savings,  are 
likely  to  occur  over  a  multi-year  period. 

If  a  project  yields  a  positive  NPV,  the  value  to  the  investor 
increases  by  the  amount  of  the  NPV.  (Conversely,  if  a  project  yields 
a  negative  NPV,  the  value  to  the  investor  decreases  accordingly.) 
Thus,  an  investment  project  is  considered  economically  attractive  if 
its  NPV  exceeds  zero.  The  decision  rules  under  NPV  analyses  are  to 
accept  all  independent  projects  whose  NPV  is  greater  than  zero  and  to 
rank  the  investments  in  descending  order  of  NPV. 

5. 2. 2. 2  Internal  Rate  of  Return 

The  Internal  Rate  of  Return  (IRR)  is  defined  as  the  interest  rate  that 
equates  the  present  value  of  expected  future  net  annual  cash  flows  to 
the  present  value  of  the  capital  investment.  IRR  is  frequently 
referred  to  as  return  on  investment  (ROI).  Mathematically, 

2) 


This  equation  is  the  same  as  Equation  5-1,  with  the  exception  that  for 
IRR  the  equation  is  solved  for  the  interest  rate  (R)  which  equates  the 
present  value  of  the  investments  to  the  present  value  of  the  net 
annual  cash  flow.  In  practice,  determining  a  project's  IRR  is  done  on 
an  iterative  basis.  In  general.  Equation  5-2  cannot  be  solved  for  R 
in  a  closed  form. 
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It  is  important  to  note  that  IRR  does  not  measure  the  rate  of  interest 
earned  on  the  initial  outlay  for  a  project.  IRR  projects  the  rate  of 
interest  that  must  be  earned  on  the  unrecovered  balance  of  an 
investment  so  that  the  initial  investment  is  exactly  recovered 
(repaid)  at  the  end  of  project  life. 

If  a  project's  IRR,  (R),  is  equal  to  the  discount  rate  or  the  cost  of 
capital  (K),  the  investor  exactly  breaks  even  on  the  investment.  When 
R>K,  the  investor  has  gained  value,  and  when  R<  K,  the  value  to  the 
investor  has  decreased.  The  decision  rule  is,  therefore,  to  accept 
independent  projects  where  R  >  K  and  to  rank  mutually  exclusive 
projects  in  decreasing  order  of  IRR. 

IRR  is  a  technique  that  is  useful  in  situations  where  knowledge  about 
future  interest  rates  is  uncertain.  Unlike  NPV,  IRR  is  calculated 
without  assuming  an  interest  rate.  Calculation  of  a  project  IRR 
allows  comparison  with  some  established  minimum  attractive  rate  of 
return. 

5. 2. 2. 3  Comparison  of  NPV  and  IRR 

The  strengths  and  weaknesses  of  NPV  and  IRR  are  treated  together  here 
because  they  are  fundamentally  identical  and,  under  identical 
assumptions,  will  generate  the  same  accept/reject  decision  for  a 
project.  The  difference  is  in  the  focus  of  each:  in  NPV,  the  stream 
of  annual  net  cash  flows  (costs  and  benefits)  is  discounted  back  to 
the  present  under  an  assumed  discount  rate,  whereas  the  same  equation 
for  IRR  is  solved  for  the  discount  rate  that  equates  the  present  net 
value  of  annual  cash  flow  benefits  to  the  present  net  value  of 
investment  costs.  In  the  context  of  economic  analyses  performed  by 
TACOM,  both  techniques  are  appropriate. 

Under  the  NPV  method,  a  TACOM  project  would  be  accepted  if  its  NPV  is 
greater  than  zero  (i.e.,  the  project  has  positive  net  present  value). 
NPV  also  is  used  to  rank  mutually  exclusive  projects.  From  the 
mathematical  formulations  presented  for  IRR  and  NPV,  if  R>K,  NPV  must 
then  be  greater  than  zero.  Accordingly,  both  approaches  will  give 
consistent  accept/reject  decisions.  However,  caution  must  be 
exercised  when  comparing  projects  of  different  size  (capital  outlay), 
duration,  or  if  the  annual  net  cash  flows  of  one  project  are 
substantially  different  from  that  of  another. 

IRR  is  applicable  even  if  the  discount  rate  is  known,  such  as  in  the 
TACOM  case  where  the  discount  rate  is  specified  by  DOD.  For  TACOM,  a 
project  would  remain  in  the  running  as  long  as  its  IRR  is  >10 
percent.  IRR  and  NPV  are  both  useful  for  their  difference  in  focus  on 
the  assumptions  in  an  investment  decision.  When  a  conflict  in  project 
ranking  using  IRR  versus  NPV  arises,  the  decisionmaker  should 
generally  rely  on  NPV  because  that  measure  indicates  actual  increases 
in  value  (benefits)  stemming  from  a  project,  as  opposed  to  IRR  that' 
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is,  in  effect,  a  derived  interest  rate  indicating  the  potential  return 
on  investment,  but  which  says  nothing  about  actual  dollar  benefits. 

One  drawback  in  relying  on  NPV  as  a  decision  criterion  for 
prioritizing  TACOM  MANTECH  projects  is  the  general  tendency  of  NPV  to 
favor  large  projects  even  though  small  projects  also  can  generate 
large  benefits.  This  tendency,  attributable  to  the  fact  that  NPV  is 
an  absolute  measure  as  opposed  to  a  derived  ratio,  can  be  overcome 
through  analytical  techniques  such  as  incremental  cost/benefit 
(profitability)  analyses. 

5.2.3  Relative  Measures  of  Merit 


5.2.3. 1  Profitability  Index  (PI) 

The  profitability  index  (PI),  a  benefit/cost  ratio, 
following: 


PI 


N 

=  Z 


N 


t=0  Cl+K)  t=0  U+K) 


is  defined  by  the 


(5-3) 


PI  shows  the  relative  profitability  of  a  project,  i.e.,  the  present 
value  of  benefits  per  dollar  invested.  Like  NPV  and  IRR,  it  produces 
a  measure  of  merit  that  can  be  used  in  ranking  projects.  The  use  of 
PI  produces  accept/reject  decisions  consistent  with  NPV  measures  of 
merit.  However,  NPV  and  PI  can  give  different  rankings  for  a  group  of 
projects  evaluated  using  both  techniques.  This  phenomenon  occurs  in 
mutually  exclusive  projects  if  differing  scales  of  investment  are 
compared.  Table  3  provides  an  example  that  compares  NPV  and  PI 
calculations;  it  illustrates  the  potential  variation  in  rankings  using 
both  techniques. 

When  using  NPV  to  rank  alternatives.  Project  A  would  be  chosen  over 
Project  B  due  to  its  higher  NPV.  However,  on  a  relative  basis,  i.e., 
benefits  per  dollar  invested  (PI),  Project  B  would  be  chosen.  To 
overcome  this  apparent  inconsistency  in  project  ranking,  it  is 
necessary  to  normalize  one  project  to  the  scale  of  the  other.  The 
classic  techniques  for  normalization  of  investment  scale  have  an 
implicit  assumption  of  profit  maximization  over  time.  This  assumption 
is  inappropriate  in  the  context  of  TACOM;  accordingly,  an  alternative 
means  of  normalization  must  be  developed  for  inclusion  in  the  economic 
evaluation  process. 

5. 2. 3. 2  Savings/Investment  Ratio 

SIR  is  identical  to  the  profitability  index,  but  SIR  is  calculated  for 
proposed  projects  whose  benefit  streams  consist  of  cost  reductions 
rather  than  increased  benefits  created  as  a  result  of  the  investment. 
Because  the  equations  are  identical,  SIR  will  not  be  evaluated  further 
in  this  section.  The  difference  in  terminology  reflects  a  focus  on 
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Table  3.  Hypothetical  Project  Cost  and  Benefit  Data 


• 

Project  A 

Project  B 

Net  Cash 

Net  Cash 

Year  PV  Factor 

FLow(Ft) 

P»(Ft) 

Flow(Ft) 

PV(Ft) 

1 

0.9091 

200.00 

181.82 

71.50 

65.00 

2 

0.8264 

200,00 

165.28 

71.50 

59.09 

3 

0.7513 

200.00 

150.26 

71.50 

53.72 

4 

0.6830 

200.00 

136.60 

71.50 

48.83 

5 

0.6209 

200.00 

124.18 

71.50 

44.39 

6 

0.5645 

200.00 

112.90 

71.50 

40.36 

7 

0.5132 

200.00 

102.64 

71.50 

36.69 

8 

0.4665 

200.00 

93.30 

71.50 

33.35 

9 

0.4241 

200.00 

84.82 

71.50 

30.32 

10 

0.3855 

200.00 

77.10 

71.50 

27,56 

1229.90 

439.31 

'i23ir.ja 

'439  ;uo 

PV  Factor  at 

10%  Discount  Rate 

Project  A 

Project  B 

Investment  = 

1000 

Investment  =  300 

Net  Present  Value  =  230 

Net 

Present  Value  = 

139 

• 

Comparison  of 

NPV  and  PI 

Results 

Project  PV 

Costs  PV 

Benefits 

NPV 

Pi 

A 

1000 

1230 

230 

1.23 

B 

300 

439 

139 

1.46 

pia 

=  1230/1000  = 

1.23 

pib 

=  439/300  = 

1.46 
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public  sector  investment  alternatives  (for  SIR)  as  opposed  to  the 

profitability  of  a  potential  investment  in  the  private  sector. 

Because  the  principal  objective  of  TACOM  in  this  respect  is  cost 

reduction  in  manufacturing  processes,  SIR  is  a  highly  appropriate 

measure  of  merit  for  use  in  the  economic  evaluation  of  proposed 
MANTECH  projects. 

In  TACOM  MANTECH  investment  alternatives,  as  contained  in  the  P-16 
exhibits  and  the  command  survey,  several  common  characteristics  of 
MANTECH  projects  were  observed.  These  characteristics  included 

•  Projects  are  usually  independent, 

•  Generally  similar  MANTECH  Program  funding  periods— usually  3 
years, 

•  Generally  similar  periods  of  time  required  to  implement  MANTECH 
projects,  and 

t  Project  implementation  generally  remains  in  force  for  12  to  15 
years. 

After  evaluating  TACOM  MANTECH  projects  (as  presented  in  the  P-16 
exhibit),  it  is  apparent  that  the  small  differences  in  project 
lifetime  will  not  introduce  a  significant  problem  into  the  economic 
evaluation  and  prioritization  of  proposed  TACOM  MANTECH  projects. 

5.2.4  Payback  Period 

The  payback  period  is  the  length  of  time  required  to  recover  the  first 
cost  of  an  investment  from  the  net  cash  flows  produced  by  that 
investment  for  an  interest  rate  equal  to  zero.  In  other  words,  the 
payback  period  is  the  minimum  time  required  for  an  investment  to 
become  profitable,  ignoring  the  time  value  of  money.  In  comparing 
investment  alternatives,  the  payback  period  measure  of  merit  indicates 
which  alternatve  will  be  the  first  to  recover  its  initial  cost. 
Mathematically,  the  payback  period  is  the  number  of  years  (N)  until: 

2  CC.  -  B.I  =  0  (5-4) 

t=Q  ^ 


where  C^  =  Costs  in  year  t:  and 
=  Benefits  in  year  t. 

Payback  also  can  be  calculated  on  a  discounted  basis  but  the  basic 
flaw,  as  noted  above,  is  still  inherent  in  the  calculated  measure  of 
merit.  As  noted  in  11-28  (Ref.  5)  ,  payback  is  considered  to  be  an 
inferior  measure  of  merit  because  it  ignores  project  benefit  and  costs 
once  the  cash  outlay  for  the  investment  has  been  recovered.  This 
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means  that  projects  with  long-term  benefits  but  low  initial  benefits 
are  less  favored  than  short-lived  investments.  In  general,  experience 
indicates  that  this  bias  is  unjustifiable  and,  in  many  cases, 
economically  unsound.  Nonetheless,  the  technique  is  simple  to 
calculate  and  intuitively  attractive  to  many  decisionmakers.  As  such, 
it  is  recommended  that  it  be  used  only  as  a  supplementary  method  to 
the  more  rigorous  techniques  (e.g.,  NPV,  IRR,  PI).  In  instances  where 
projects  yield  identical  estimates  of  PI,  IRR,  NPV,  or  SIR,  payback 
can  be  used  as  decision  criteria  to  determine  which  investment  should 
be  made  on  the  basis  of  most  rapid  cost  recovery. 

5.3  Comparison  of  Measures  of  Merit 


5.3.1  Data  Requirements 

NPV,  IRR,  PI,  and  SIR  data  requirements  are  identical.  Information 
necessary  to  analyze  projects  based  on  these  measures  of  merit 
includes:  annual  net  cash  flows  (net  benefits),  project  capital 
investment  cost,  discount  rate  as  specified  by  DOD,  and  the  number  of 
years  in  the  lifetime  of  the  investment  plus  the  life  cycle  length  of 
the  project  as  implemented.  Payback  period  requires  only  the  flow  of 
annual  net  benefits  and  costs  up  to  the  time  that  the  two  are  equal. 

5.3.2  Ease  of  Calculation 


Again,  there  is  very  little  difference  in  the  ease  of  calculation 
between  the  four  primary  measures  of  merit  listed  above.  Because  the 
determination  of  IRR  is  an  iterative  process,  its  calculation  is  the 
most  time  consuming.  However,  with  the  widespread  use  of  computers, 
large  data  sets  can  be  manipulated  easily  and  efficiently  using  any  of 
these  measures  of  merit.  Calculation  of  payback  period  is  the 
simplest  measure  to  calculate,  and  this  is  a  primary  reason  for  its 
attractiveness  to  decisionmakers.  However,  it  is  crucial  to  keep  in 
mind  the  caveats  associated  with  it. 

5.3.3  Unequal  Project  Lifetimes 

In  comparing  independent  project  investments  (both  private  and  public 
sector),  the  analyst  is  often  confronted  with  project  alternatives 
having  differing  economic  lifetimes.  When  comparing  such 
alternatives,  it  is  necessary  to  equalize  or  normalize  project 
lifetimes  so  that  all  alternatives  are  evaluated  on  a  comparable 
basis.  This  is  usually  accomplished  by  adopting  the  longer  lived 
project  lifetime  as  the  economic  lifetime  and  repeating  the  shorter 
lived  investment,  i.e.,  replacement  investing,  so  as  to  allow 
comparable  periods  of  investment  for  analysis. 

5.4  Conclusions 

The  measures  of  merit  considered  in  this  report  range  from  overly 
simplistic  (e.g.,  payback  period)  to  fairly  sophisticated  treatments 
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of  costs  and  benefits.  Table  4  summarizes  the  measures  of  merit 
evaluated.  Based  on  the  range  of  measures  evaluated,  several  highly 
desirable  characteristics  of  an  ideal  measure  of  merit  for  TACOM  use 
are  identif i cable : 

•  It  must  have  a  discounted  cash  flow  basis, 

•  Both  absolute  and  incremental  aspects  of  an  investment  should 

be  reflected  in  the  measure  of  merit, 

•  It  should  be  simple  to  calculate,  and 

•  Its  meaning  should  be  clear  and  not  subject  to  multiple 

interpretation. 

All  the  measures  of  merit  considered  in  this  report  are  prone  to  at 
least  one  common  weakness— al 1  require  the  estimation  of  future 

benefits.  Investment  costs,  in  the  context  of  proposed  MANTECH 
projects,  are  fairly  straightforward  to  estimate  or  are  at  least 
given,  i.e.,  bounded.  Benefits  are,  however,  far  more  subjective  and 
speculative  because  they  are  an  integral  element  in  all  the  measures 
considered  and  are  most  likely  to  be  a  source  of  potential  error  and 
subsequent  misestimation.  Thus,  the  remaining  work  of  Task  III  will 
emphasize  the  development  of  guidelines  for  determining  benefits. 

The  various  techniques  discussed  here  can  be  ranked  in  terms  of  the 
desirable  characteristics  listed  and  usefulness  to  TACOM  in  the 
evaluation  and  prioritization  of  proposed  MANTECH  projects.  Ranking 
from  most  useful  to  least  useful,  they  are: 

•  PI  or  SIR,  as  appropriate; 

•  NPV; 

•  IRR;  and 

•  Payback. 

Both  the  PI  and  SIR  measures  of  merit  come  the  closest  to  satisfying 
the  needs  of  TACOM  in  terms  of  an  economic  measure  of  merit  to  be  used 
for  project  ranking.  They  are  incremental  in  nature  and  reflect  the 
cost  effectiveness  per  dollar  expended.  Payback  period  is  not 
recommended  for  TACOM  use  because  it  does  not  factor  into  the  analysis 
any  consideration  of  project  cash  flows  after  the  initial  investment 
payback  period  is  achieved. 

Despite  the  scope  of  the  economic  measures  of  merit  evaluated,  there 
are  at  least  two  additional  factors  that  must  be  reflected  in  the 
evaluation  and  ranking  process;  they  are: 
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Discount  rate  (TACOM  is  directed). 

Annual  costs  attributable  to  tproject  investment. 


•  Economic  and  technical  uncertainty  and  risk  and 

•  Benefits  that  are  nonquantif iable  in  economic  terms. 

Economic  uncertainty  can  be  accommodated  by  producing  estimates  of 
high,  low,  and  most  probable  levels  and  timing  of  costs  and  benefits. 
When  combined  with  estimates  of  probability  for  each  of  the  three 
levels,  probabilistic  estimates  of  PI  or  SIR  can  be  developed.* 
Technical  risk  can  be  accommodated  in  a  similar  fashion.  That  is, 
estimates  of  high,  most  probable,  and  low  degrees  of  technical  risk, 
together  with  the  probability  of  their  occurence,  can  be  developed  and 
incorporated  into  the  evaluation  process. 

Benefit  types  (project  outputs)  that  resist  straightforward  expression 
in  terms  of  cost  should  be  reflected  in  the  overall  project  evaluation 
process  despite  the  fact  that  some  outputs  are  exceedingly  difficult, 
if  not  impossible,  to  express  in  dollar  terms.  AR  11-28  (Ref.  5) 
states  that  benefits  in  this  general  category  should  be  presented  with 
the  economic  evaluation  so  that  estimates  of  project  output  (i.e., 
indicators  of  objective  achievement)  are  available  to  the 
decisionmaker.  Further,  difficulty  or  inability  to  quantify  outputs 
is  not  a  basis  for  disregarding  output  analysis.  These  benefits 
should  be  included  when  analytically  and  economically  feasible. 
Heightened  readiness  capability  is  an  example  of  this  type  of 
problem.  It  may  not  be  readily  expressed  in  terms  of  dollar  benefits, 
yet  the  estimated  percentage  increase  in  readiness  and  the  necessary 
investment  can  be  presented.  Such  estimates  have  material  value  in 
that  they  provide  the  decisionmaker  with  at  least  some  basis  for 
determining  the  allocation  of  limited  resources. 


♦Taken  one  step  further,  sensitivity  analyses  can  be  conducted  to 
address  economic  uncertainty.  These  analyses  involve  focusing  on 
the  assumption(s)  in  the  analysis  (usually  benefits)  about  which  the 
least  is  known.  That  particular  benefit  or  cost  is  varied  by  +5  or 
10  percent,  for  example,  and  the  measure  of  merit  is  recalculated. 
These  evaluations  reveal  the  sensitivity  of  a  project's  NPV  (or 
other  measure  of  merit)  to  misestimation  of  a  variable  that  also  can 
be  accotmiodated  by  project  evaluations.  That  is,  estimates  of  high, 
most  probable,  and  low  degrees  of  technical  risk,  together  with  the 
probability  of  these  occurrences,  can  be  developed  and  incorporated 
into  the  evaluation  and  project  ranking  process. 
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LIST  OF  ACRONYMS  AND  ABBREVIATIONS 


AFSC 

AVSCOM 

AFWAL 

AMCCOM 

AMMRC 

ASD 

BMO 

CAM 

COCO 

DARCOM 

DCF 

DDR&E 

DMT 

DOD 

ERA 

FYDP 

GOCO 

GOGO 

IBEA 

IRR 

MAC  I 

MANTECH 

MI  COM 

MMT 

MTAG 

MTBF 

MTTR 

NAVMAT 

NMCIRD 

NPV 

OMA 

OSHA 

PA/OMA 

PEM 

PEP 

PI 

PM 

R&D 

RAM-D 

RDT&E 

RFP 

SD 

SIR 

TACOM 

TC 

TECHMOD 


Air  Force  Systems  Command 

Aviation  Systems  Command 

Air  Force  Wright  Aeronatucial  Laboratories 

Armament,  Munitions,  and  Chemical  Command 

Materials  and  Mechanics  Research  Center 

Aeronautical  Systems  Division 

Ballistic  Missile  Organization 

computer-aided  manufacture 

contractor  owned  contractor  operated 

U.S.  Army  Material  Development  and  Readiness  Command 

discounted  cash  flow 

DOD  research  and  engineering 

Directorate  of  Manufacturing  Technology 

U.S.  Department  of  Defense 

Environmental  Protection  Agency 

five-year  defense  plan 

government  owned  contractor  operated 

government  owned  government  operated 

Industrial  Base  Engineering  Activity 

internal  rate  of  return 

Military  Adaption  of  Commercial  Items 

Manufacturing  Technology  Program 

Army  Missile  Command 

Manufacturing  Methods  and  Technology 

Manufacturing  Technology  Advisory  Group 

mean  time  between  failure 

mean  time  to  repair 

Naval  Material 

Naval  Material  Command  Industrial  Resources  Detachment 

net  present  value 

operations  and  maintenance.  Army 

Occupational  Safety  and  Health  Administration 

procurement  appropriation/OMA 

Production  Engineering  Measures  Project 

producibility  engineering  and  planning 

profitability  index 

Project  Manager 

research  and  development 

reliability,  availability,  maintainability,  durability 

research,  development,  testing,  and  evaluation 

Request  for  Proposal 

Space  Division 

savings/investment  ratio 

U.S.  Army  Tank-Automotive  Command 

technical  completion 

technology  modernization 
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